http://www.techspot.com/review/849-intel-pentium-anniversary-edition-overclock/
Intel is now selling a cheap unlocked dual core CPU that can be used building a cheap gaming PC. The article's useful in providing some benchmarks to see what's possible, & not, but their idea of cheap is to me a bit clueless -- it may be that they just went the safest route [being really cheap does require some research & discretion], or they've never had to pinch pennies as they say. Eliminating the SSD, sticking with their choice of motherboard, CPU heatsink, & graphics card, I quickly come up with $200 in savings -- it's very possible you could save more, e.g. going with a cheaper graphics card model or if you found one on sale. You could also save another $20-$30 with a reman rather than new hard drive, maybe $50 going with a different motherboard on sale, & almost $30 by sticking with the OEM CPU heat sink. Do that & I think it's entirely possible to wind up with a price tag around $350 less than their estimate. Newegg has a this new CPU bundled with a micro ATX motherboard for $83, knocking another $32 off on top of that proposed $350 savings.
How well that PC would stack up against a complete PC you could buy depends on the components used in the complete PC, & of course on its sale price. It really could go either way. There's no question a decent sized company can buy in enough quantity to get great pricing on the parts that make up a PC, and there's no question that they can afford to reduce their profit margin per component because they're selling you all of them at once. And don't forget, the full PC comes with a copy of Windows that you'd have to buy if you built your own. That said, any company that builds PCs may put total junk inside those cases. The only way to tell, to make a fair comparison, is through researching any make/model you're thinking about buying.
The new CPU is the dual-core G3258 -- Newegg has it for $70, on sale for $65... hard to say with this one, but if it's like most CPUs in the past, sooner or later it'll be on sale for closer to $50. Being unlocked means that you can try setting it to run faster than it's designed for -- more on that aspect later. Some games rely on the graphics hardware MUCH more than the CPU -- in those cases the new chip might work well enough, saving you some cash, but not so much for software that's designed to use 4 or more cores. On to the main PC components...
Power supplies can be out & out frauds -- they can include fake electronics components, e.g. a shell of one placed over a much less expensive alternative, or even shells filled with rocks or pebbles or concrete. They can do a great job of highly regulating voltages & current, operating very efficiently, or they can do the same jobs very poorly, & you can't go by price. You can probably get away with a lower performing power supply, but why bother -- you can research to find good performing models that have a good record of longevity, & at least in the US, find one on sale fairly cheap, maybe $30-$40 for something in the 600-700 watt range that you'd want for a nicer graphics card. The power supply wattage you need is based largely on the graphics card(s) you want to power -- there are web sites with calculators.
A PC case is a thin metal shell to hold everything -- as long as it has adequate provisions for cooling, the cheapest case you find [for maybe ~$20-$30] will work just as well as a case you'll pay $200 for. You can pay extra for lighter weight aluminum, or for a design you think looks prettier. You can pay extra for sturdier construction -- that only matters if you'll move it &/or take it apart a lot, & is pretty much irrelevant if it always stays in the same spot on top of or under your desk or table. You can pay extra for a bigger case -- I find that handy as my hands don't work as well as they once did, because there's more room if you fumble. You can pay extra for a case that makes it easier to replace or add components, like hard drives or add-on cards, but if you're not going to be doing that stuff once it's together, why bother? More expensive cases come with more fans -- a good but cheap case has places you can add fans, but the fans themselves are missing. You can most always buy better fans than anything that comes with any case -- by better I mean more air flow &/or less noise.
CPU cooling is a PITA. You can go with air cooling, meaning a heat sink & fan(s), & have to keep it cleaned of dust bunnies. You can go liquid cooling, but then have to clean a radiator. I lean towards a closed liquid cooling system because they're often on sale, & because heat sink fins can be sharp enough to cause cuts, and because it's easier to have air flow across the radiator with its included fan(s) than it is to ensure a lot of airflow past a heat sink. On the down side there's much more to go wrong with liquid cooling, & at the very least it will wear out -- a heat sink is basically a lump of metal that can only get dirty, & the worst that can happen is replacing the fan(s).
Now, overclocking... It started because it was possible -- some people make their cars go faster, & some people like to make their PCs go faster, at least in part by having the CPU run at a higher frequency. It became popular because manufacturers run their electronics through quality control, selling the stuff that fails for less $, only they didn't [don't] always have enough failures to fill the demand for lower cost versions. They try to make sure no one can unlock the full potential of those lower cost CPUs, with varying degrees of success. They also sell unlocked CPUs, usually for a higher price.
Overclocking is done in the motherboard bios settings -- some boards have more options, features, & capabilities in this regard than others, & as you'd suspect, more options = more $. The chief downside is that overclocking increases temps, assuming that you limit your overclocking to the max speed where the PC is stable, runs & is not prone to crashing. The best way to overclock is found researching whatever brand/model of motherboard. Research will also tell you what to expect from most CPUs, though this is one category where mileage can definitely vary, & there are few if any absolute guarantees.
Many people also overclock their graphics card, & many cards come overclocked from the factory.