LDO Regulators (Low Drop Out Regulators) are cheap and abundant, but require a bit of knowledge to implement. As mentioned for solar panels < ~10W a simple zener will prevent over charging. Replacing a relay with an SCR or FET will save you some power. Going to lithium-ion batteries or even a capacitor bank will eliminate the need to worry about deep discharge cycles. You might consider a "timer" that "turns on after dark" and then stays on for some preset period of time (say 6 hours).
For LED based projects and small solar panels capacitor storage is becoming a reality. With capacitors in the 2.3V to 5.5V range values of 5F to 300F are becoming readily available and getting cheaper every day. I will make a post with the math and what it means when I get some time, but the bottom line is that capacitors hold a lot of promise to the "Green LED lighting" crowd; virtually no losses, long life and tons of potential.
Oh heck, let's do it real quick....
Q = CV
U = 1/2CV^2
Q = Charge in Columbs
C = Capacitance in Farads
V = Voltage in Volts
U = Energy in Jules
If we have an LED rated for 2.3V and 30mA how long could we light it with a 2.7V 200F capacitor? LED voltage ratings and current ratings are really a complex, non-linear relationship. The short version is that an LED will draw more current than 30mA if placed across a 2.3V source. The amount of light an LED will output is also a non-linear function of the current. In practice current is limited with a series resistor for a nominal voltage. If we assume 90% of the LED's light output will be achieved @ 20mA (a reasonable assumption) and our maximum voltage will be 2.7V, then our series resistor would need to be:
2.7V-2.3V = 0.020A * R ==> R = 20 ohms
Our capacitor, when fully charged to 2.7V will have:
U = 1/2 * 200F * 2.7^2 = 729J
Assuming the LED will dim considerably @ V < 2.0V, the energy in the capacitor @ 2V will be:
U = 1/2 * 200F * 2^2 = 400J
The amount of useful energy would then be 729J - 400J = 329J
Our LED power consumption should average ~ 2.3V * 0.020A = 46mW
Watts are defined as Joules/Second, so 46mW would be .046J/Second, so 329J/(.046J/Second) = 7152 seconds or roughly 2 hours of "full brightness". This is actually a conservative estimate and assumes "constant power to the LED." A simple switching circuit that turned the LED on and off could increase the run time by as much as a factor of 3 or 4 (6 to 8 hours).
Now, if we took two of these 200F 2.7V capacitors and placed them in series (now 100F @ 5.4V) and re-calculated:
5.4V - 2.3V = 3.1V 3.1v = 0.020 * R ==> R = 155 Ohms
U = 1/2 * 100 * 5.4^2 = 1458J
U = 1/2 * 100 * 2^2 = 200J
1458J - 200J = 1258J
1258J/0.046W = 31,695 Seconds ==> 8.8 hours.
Again, there are numerous approximations involved, a formal analysis would take into account the power lost in the resistor, and the change in power as the voltage decreased over time, but I think you get the idea. In both cases the use of an active electronic current controller would greatly extend the run time.
It is important to note that the energy stored in a capacitor is a function of the square of the voltage. Obviously high energy density favors higher voltage. A 200F 2.4V capacitor is capable of storing the same amount of energy as a 2F 24V capacitor (576J)! It is interesting to note that two equal capacitors placed in series have 1/2 the capacitance of either capacitor alone; however, the energy stored by the two in series => C * V^2 this is an increase of 50% (that is, as you would expect, there is twice the energy stored in two capacitors than there is in one). What changes is the amount of USEFUL energy that is stored. The trick is to use an active circuit to utilize a wide range of capacitor voltages with particular attention to the "upper voltage region" where the highest energy density resides.
One simple circuit to maintain "constant brightness" over a wide voltage range is to use PWM (Pulse Width Modulation". Lets say we have a "base frequency" of 100hz (the LED turns "on and off" 100 times a second). Using a simple active circuit, we can vary the length of time the LED remains in the "on" cycle based on the supply voltage. At high voltage the LED may have a duty cycle of 1% (or 0.01 * 0.01 = 100uS). Toward the middle of the Voltage range it may have a duty cycle of 50% (or 0.01 * 0.50 = 5mS). Near the end of the Voltage range it may have a duty cycle of 100%, or 10mS. Without wasting power in a resistor (power loss in the resistor = I^2 * R), the "apparent brightness" of the LED can remain constant over a defined voltage range, thus extending the run time considerably.
Even a few years ago a 10F capacitor was a very expensive piece of lab equipment, but the advances in capacitors has led to 10F to 1000F capacitors relatively cheap and easy to find. As 100F to 10,000F capacitors in the 10V to 50V range become available, storing solar energy in them will emerge; how we utilize that energy needs to be as carefully thought out as collecting and storing the energy, and LED lighting is an area well-suited for the task.