This question comes from Genesis 4.3-7 where Cain, the firstborn of Eve and Adam, takes an offering of his crops to the Lord and Abel took a first-born lamb. God accepted Abel's offering, but not Cain's. In the course of the story Cain becomes angry and was apparently saddened, but God offers no explanation of why the offering wasn't accepted. Instead God says to Cain, "Why are you angry, and why has your countenance fallen? If you do well, will you not be accepted?" (Genesis 4.6-7). In the end, Cain's anger and jealousy gets the best of him and he murders his brother Abel.
So, why was Cain's offering not acceptable?
We turn to both the Old and the New Testaments to find our answer. Cain and his deed is mentioned three other times in the Bible, outside of the Genesis story. The writer of Hebrews attributes Cain's lack of faith as the reason for God's rejection of the offering (11.4). John attributes Cain's acts as a result of his evil disposition (1 John 3.12). Finally, Jude implies the offering was rejected because Cain's motivation was greed (1.11).
The complete answer is alluded to in the Genesis passage. In Genesis 4.3-4 we read, "In the course of time Cain brought to the Lord an offering of the fruit of the ground, and Abel for his part brought of the firstlings of his flock, their fat portions." Here Abel brings an offering of, what would later be called, "first fruits" while Cain brought simple an "offering of the fruit of the ground." Implied here is that Abel obeyed the yet non-existent law, while Cain did not.
The laws of giving a first fruit offering are outlined in Leviticus, Numbers, and Deuteronomy. The essence of the law is that the first fruits of any harvest, whether it be grain, fruit, or animal is to be offered to God and brought to the priests (Deuteronomy 18.4). The law is clear that the first fruits are just that, the very first born, first harvested, etc. They are considered the best of the best (Numbers 18.8-14). Apparently, Abel brought this sort of offering, since he brought the "firstlings of his flock"; however, Cain did not. This is likely what made Cain's offering unacceptable he brought God the "left-overs" instead of the first and the best.
But wait how was Cain to know what God required, since Moses hadn't been born and there was no law yet?
Two answers. One: God deserves the best period. Cain should have known that, as should we all.
But there's a second answer that takes into account the history of our scriptures. First, we need to recall that the majority of the books of the Old Testament weren't put into writing until the Babylonian exile, some 400 years before Christ. During that time the institution and the study of the Torah, the law, had become important and many of our biblical stories were written to illustrate the importance and/or the origins of individual laws. These stories are called etiologies. It is likely the Cain and Abel story is an etiology illustrating the consequences of not giving God the required first fruits offering. And though the laws of the Torah were not available to Cain, for the sake of the illustration this point was irrelevant.
Perhaps Cain's offering wasn't accepted because he had a lack of faith (Hebrews 11.4), for certainly his "faith" wasn't in full practice with his disregard for giving God the best. Perhaps his offering wasn't accepted because he was filled with greed (Jude 1.11) or because he had evil intentions (1 John 3.12). Whatever his motivations, the story of God's rejection of Cain's offering is probably written because the writer wanted to show the consequences for disregarding the laws of God, especially the law of offering to God what belongs to God (Matthew 22.21).
