How is it done?

[justDIY]

Found this on del.icio.us:

http://mrl.nyu.edu/~jhan/ledtouch/

Dude is using an led matrix display as some sort of interactive touch sensor

I've watched the video again and again, but I haven't figured out how he's using the led as a sensor and an emitter at the same time?

[ReefRaff]

Thats pretty cool. Maybe HE is building a mars probe. No email either or we could write and ask him.

I wonder if by passing your finger over the led causes it to heat up more. Maybe he built a detector to detect the increased heat over any given LED.  just a thought.

[ReefRaff]

better yet. Could increased heat over a LED cuase it to draw more or less amps? maybe thats his detector?

[SurJector]

He writes that the led is a photovoltaic device. I think he uses the reflection of the light on the finger which modifies the electrical characteristics of the leds (that's what was guessed by Rob in the "auto-illumination" topic). I don't know if it's measurable with an ordinary voltmeter. I've to test it tonight !

[xraycatj]

sweet, I bet that could be used with a synthesiser or something for making music......

[justDIY]

He writes that the led is a photovoltaic device. I think he uses the reflection of the light on the finger which modifies the electrical characteristics of the leds (that's what was guessed by Rob in the "auto-illumination" topic). I don't know if it's measurable with an ordinary voltmeter. I've to test it tonight !

well, a quick read of some mind-numbing scientific articles on the subject, and I said the heck with this engineering hoopla, time for some trial and error (i did business in school not engineering)

I first started playin with a yellow led, as it was what I had laying around.   it worked real well with a white flashlight ... I saw ~1.5v across it when shining it with my 4d maglight

but when I switched to a little laser diode i tore from a level, I got bupkiss... so flashing back to all the engineering hoopla, I remember reading that leds have a narrow 'band gap' and they're only sensitive to frequencies they emit and higher ... so, a red laser shining into a yellow led just wont work.

eventually i dug out a red led, and it works very well with the red laser and the flashlight.  well this was all well and good, but the amperage the led is probably in the micro-amp range, thanks to the high impedence of my dmm, the voltage is easily visible, but I wasn't able to drive one led from another led, not even a faint glow.   so I tried hooking the led upto the ADC on my microcontroller ... well I was suprised - there was a reading... with the laser point-blank into the led, I read 1.56v, however, with anything less that total saturation of the led, the reading was all over the place on the ADC... I dont know what causes that or how to fix it, so I kept pluggin along.

I was reading that it was recommended to use an OP-AMP to boost the signal from the led into something with more kick to it... so, I dug around a bit, the only op-amp I could find was the lm386 ... a low wattage audio amplifier... so I figured, what the heck, I hooked the LED upto the op-amp's in+ and in- and the out was connected to my dmm and the ADC ... well hot damn, there's a more stable signal!  it measured about 2.6v in the dark and ~4.5v at full saturation with the laser or maglight (it didn't really seem to make a difference)

not really sure what to do now, but I'm glad I was able to make something happen.   That guy's system must have some an ADC with a lot more channels or he scans the array somehow... and I'm still wondering about the way it seems his led matrix is on, but he's able to get readings from it.   I wonder if he's turning each led off breifly and measuring the voltage it generates based on reflections from the leds surrounding it.  I have 8 channels on my ADC, but only have the one op-amp... next order i make somewhere, maybe I'll get one of those led matrix and a proper op-amp (with multiple amps) - ahh another project to consume my time :/

[cpemma]

If you cheat and use a 3-wire double led (I've got some such yellow ultrabrights, also seen both common-anode and common-cathode bi-colors) then Bill Bowden's circuit would work. One led as light, other as sensor. 

[justDIY]

ahha, so the video might be using one of those bicolor or rgb led matrix - well no, that wouldn't work

if the matrix were blue or green, I could see using the red as the detector, but it looks red in the video, which wouldn't be detected by blue or green leds

the secrets continue...

[justDIY]

well, I'm obvisouly not the only one intrigued by that led motion sensing video

just off the link-wire is a technical paper from some scientists at Mitsubishi ... very low cost sensing and communications using bi-directional leds (by bi-directional, they don't mean bi-color, just that leds can send and then receive)

http://www.merl.com/publications/TR2003-035/

update:

*WOW* that is a good read ... the stuff about LEDComm and iDropper really has the 'ol brain gears a spinning with the possibilities.

[justDIY]

HAHA 

It works!!

Using my microcontroller, I whipped up a small simple program... to the naked eye, it looks like my red led is solid on ... but actually it is pulsing so fast, you cannot tell it is actually spying on you!

I set the program to light a second led when the 'sensor led' detects light ... shine my spy led from across the room with my little laser - presto the indicator led turns on!

this is just simple logic operation - is there light or isn't there light.   To take it to the next level, as seen in the video, the microcontroller would have to do a quick analog - digital conversion, reading the actual "brightness" of the light detected.

funny side effect, when I have my overhead lights on (two 40 watt fluorescents), the indicator light blinks on and off (kinda slow), obviously some sub-harmonic of the ~40 kHz the electronic ballast driving my fluorescents operates at.

obvisouly, the led cannot detect its own light, using a mirror to shine it back into itself, guess the speed of light is the limiting factor there... or maybe the speed of my microcontroller ... theoretically, light that leaves the diode as it is shut down still has a round trip time to make it out of the lens, through the air, through the glass, bounce off the silver plating, back through the glass, back through the air, back through the lens and into the diode ... so if a microcontroller could 'poll' the diode fast enough, it should be able to see its own reflection - the math involved in figuring out how fast that would need to be is way over my head

[justDIY]

Here's the code I'm using on my microcontroller, incase anyone is interested in following along:

This is in Proton Basic, for a Microchip PIC 18F452 running at 20Mhz

It should work on any pic, you may need to fudge the ports some - I'm not sure what the effect of a lower clock speed would be.

Code:

                 DEVICE 18F452
XTAL 20

TRISD = 0   ' port d initially all outputs

Symbol LED = PORTA.0    ' Onboard Status LED

Symbol Sens1 = PORTD.7 ' Sensor LED1 Anode Port
Symbol Sens2 = PORTD.6 ' Sensor LED2 Cathode Port

Symbol Dir1 = TRISD.7 ' Sensor LED1 Anode Port Mode
Symbol Dir2 = TRISD.6 ' Sensor LED1 Cathode Port Mode
 
 
Dim SensStat AS BYTE ' Sensor LED1 Port Status Variable

Clear        ' Clear everything

LOW LED        ' Onboard LED off

DelayMS 100       ' wait for things to settle

GOTO Start ' skip led test

LEDTest:

  ' just turn the led on
DIR1 = 0    ' output
Sens1  = 1    ' source

DIR2   = 0    ' output
Sens2  = 0    ' sink

DelayMS 2000    ' wait 2 sec

Sens1  = 0 ' sink

DelayMS 2000


Start:

  ' just turn the led on
DIR1 = 0    ' output
Sens1  = 1    ' source

DIR2   = 0    ' output
Sens2  = 0    ' sink

DelayMS 1    ' wait 10ms

' now reverse bias the led, charging it up
Sens1  = 0    ' sink
Sens2  = 1    ' source

DelayUS 10

'DelayMS 5    ' one ms should be plenty of time to charge the diode

DIR2   = 1    ' switch pin to input

DelayMS 1    ' wait

SensStat = Sens2   ' read the state of the pin

         SensStat = SensStat ^ 1   ' invert the result

LED = SensStat        ' set onboard led to the state, on = light was detected, off = no light

GOTO Start    ' repeat


END

[justDIY]

more updates:

after re-reading the article a few times, I realized what exactly was going on (something that wasn't all that clear at 1am!)

the led acts as an light dependent capacitor ... reverse biasing the led charges it up ... then the amount of light it is being exposed to when you stop charging and start discharging the led effects the voltage and rate of discharge.  since the PIC has a rather high impedence on its inputs, the time it takes to discharge is actually measurable.   the compiler I'm using has a function called RCIN which measures the time it takes to discharge a resistor-capacitor network.  Well, the led is sort of an RC network, and the command actually produces values which increase as the light the led is exposed to decrease.   I'm not sure that the values indicate an accurate measure of time, but the fact that I received a low value when the laser was point blank at the led, and a high value when the led was exposed to just ambient room light is enough for a simple photometric response.

so I added a second led ... and set both to alternate as input and output ... watching the stream of numbers it was clear they were working in a true bidirectional fashion, the input being illuminated by the output, alternating so fast that both were producing roughly the same 'brightness' values ... the numbers for my fingers were much higher (less reflection) than using a white piece of paper, and the numbers from the laser diode were much much lower than the white paper, indicating extreme brightness.

i have some more work to do with this stage, as right now I'm using a 'software' timer, which suspends all the other activities of the pic while its waiting for the led readings ... which results in the leds slowly alternating like a railroad crossing sign, until you interact with then, and the rate of the alternation increases since the discharge rate of the leds is being sped-up from the reflected light.   that in itself is kind of a neat "toy" ... the question to be posed would be "how is it when you pass your hand over the lights, they blink faster, pass over them with a piece of white paper or a mirror and they blink very fast, yet there is no sensor eye, no capacitive-change detector, no radio proximity detector?"

anyway, I need to look into handling the 'timing' of the discharge in a much faster manner, so even a dark led can be read in a millisecond or two so it's 'invisible' to the user.

[justDIY]

ok, well, timing the discharge of even a small capacitor (led) into a high impedance input takes too long ... in the dark, the leds were overflowing the counter, taking more than 130 msec to discharge, which obviously distorts the 'always on' disguise.

so, plan b ... instead of measuring the rate of discharge, I instead measure the voltage potential ... using a 1ms sample time, I found a decent dynamic range in 'brightness sensitivity' while still maintaining the 'always on' disguise.   Measuring the voltage is a little tricky, and somewhat noisy, but it did produce the desired effect. 

Current methology:

1>
Forward bias the led using a one I/O Port set to high and one port set to low (anode = high = +5v cathode = low = 0v). This constitutes the "ON" state of the led

2>
Now, instantly (that is with no programmed delay, so instant at 20mhz is pretty quick), reverse bias the led, by switching the values on the I/O ports (anode = low = 0v, cathode = high = 5v) This constitutes the 'CHARGE' state of the led, filling it with a 5v charge which it discharges internally based on the light level

3>
Last, again without any programmed delay, switch the cathode I/O port into high-z analog mode and connect it to the ADC input and do a 1ms sample.  I was worried about this step, since I didn't know how the pic would react to having the ports connected and disconnected from the ADC abruptly ... but it handles it flawlessly as far as I can tell.  It is an awesome feature in the pic that allows from some pins to be 'multitasked';  I'm using PORTA.0 and PORTA.1 as my dual-purpose ports, they function as digital outputs and analog inputs, and can switch between those roles very very fast.

4>
Forward bias the led again, and repeat steps 2-4 for the next led in the matrix.

-----

What I was most concerned with was reading the leds further into the matrix, those after the first led ... since I want the leds surrounding it to remain lit.   What I didn't know until I read the datasheet a few times, in Analog mode the PIC only alters ports set as inputs ... so even with the ADC running, the forward biasing of leds surrounding the led being measured is sustained.  And the same holds true for the led about to be measured, when it is time to reverse bias 'charge' the led before measuring it, keeping its ports set to output until the last instant keeps it in the desired state.

Another 'cool' part about this program, its extremely simple and very small ... the largest assembly code segment produced by the compiler is the section that deals with the serial output, dumping the numeric ADC data to my host pc for analysis.  The rest of the code is dedicated to manipulating bitwise registers inside the pic, for managing pins states, even the ADC sampling is done by hardware within the pic, all you do is tell it what pin to sample, tell it to start, and keep checking for a "done" bit to be set, and then retrieve your number from the sample register ...  since this code is so small, it executes extremely fast.  The longest delay is the 1msec sample from the ADC ... I have tried lowering the sample time (my compiler docs recommend 50 to 100 µS, however, this doesn't give the led much time be discharged after switching from being reverse biased ... so the numbers barely move.

Tonight I want to accomplish two things ... first is to write a small visual basic program which will turn the stream of numbers into bar graphs, and second, add more leds to my matrix ... two leds is quite effective for testing, but not practical for much of anything ... but I also have to figure out a scalability problem ... my pic (which is one of the most feature-packed of microchips' 8bit line-up) only has eight ADC channels ... there are some that offer up to fourteen channels, which could build a 3x4 matrix ... nothing close to the size seen in the video.

if anyone made it this far through my ramblings, congrats!

So I'll have to look into how I could use a bi-directional multi/demultiplexer (74HC4051) to expand each of the adc channels into eight more channels, giving me 64 channels, enough for an 8x8 array ... jeeze thats not even very large ... I suppose if I upgrade my pic to the biggest 18F series my protoboard can support, I'd have thirteen adc channels ... 13 x 8 = 104, so an array 10 by 10 *pout* I wonder how to get more channels, without making something terribly complex (not that driving thirteen multiplexors wouldn't be complex enough!)

perhaps slaving together a chain of smaller pics (funny, small pics offer fourteen adc channels), each handling a line of the array, with a larger pic coordinating them ... with fourteen pics, that would give an impressive array of 14x14 or 196 pixels

jeeze I just keep digging this hole deeper!  getting enough ADC inputs is the biggest challenge to making this work.  And another problem is enough ports for the led anodes as well, since wiring a matrix as common anode would cause an entire 'line' to go dark when I went to reverse bias a single led, that wouldn't provide optimal illumination of the finger / pointer that is interacting with the array, although maybe it would be enough

[SurJector]

ok, well, timing the discharge of even a small capacitor (led) into a high impedance input takes too long ...

Did you try to provide a resistor through which the LED could discharge ? I'd try to

• either put a resistor between the anode and ground and put the anode to ground while measuring the cathode (it will probably not work because the current needs to flow "backward" through the 0V);
• or put a resistor in parallel with the LED. The value is a little bit difficult to guess, because you'd like something high to not consume too much power and something low enough to allow for the discharge.

Keep on the good work !

[justDIY]

cool - thanks for the reply ... I'd been worried that this stuff was just scaring everyone / or giving them headaches!

I did read some theory about using a resistor, to measure the current stored in the led... according to what I read you place a resistor as a short, across the led, and then measure the voltage across the resistor, then using I=V/R it will tell you the current being generated by the led

the problem with this is, if you want to also use the led as an indicator, your resistor needs to be high impedence so it won't short out the led... and then you are back to square one.

The led already has a low value series resistor (150 ohms) to limit current being drawn while forward biasing the led... this resistor does not have any effect on the led as a photocell however, since the current involved is so low and I'm only interested in the voltage and not the current.

-----

I managed to get my visual basic program working last night - it gives me a running 50 sample line graph of the voltages for my leds ... the program was a little trickier to make work than I had thought - figuring out a reliable way to keep the serial data spewing from the pic synced with the reader on the pc was a challenge - and visual basic is of course, awful slow when it comes down to anything... so I've had to slow my output rate on the pic to 100 msec to keep from overflowing the buffer on the pc

-----

Picked up a quad channel op-amp last night, of course they only had one.   I'm hoping the op-amp will give me more sensitivity since the leds are not very sensitive to their own light reflected from my fingers ... when I use a white piece of paper I get a lot better results.  I've also noticed (when I was more awake) that a 1ms sample time is causing a slight flicker of the leds... but reducing the sample time is causing terrible results in the sensitivity department

I'm sorta confused as to how I'll shoe-horn this op-amp in, putting it directly between the led and the pic will prevent the pic from being able to bias the led forward or reverse, that is unless I use a third control line from the pic... for each led

line 1 = adin ... connected to the output of the op-amp
line 2 = digital ... connected to the cathode on the led
line 3 = digital ... connected to the anode of the led

led cathode connected to op-amp in+ (yes, backward, i think)
led anode connected to op-amp in-

then I forward bias, reverse bias and then go high-z on both digital lines, leaving the led to discharge into the op-amp, sample the adc real fast, and then resume forward bias

whatcha think?