Timesquare Watch and BOD-disable

For RTC breakouts, etc., use the Other Products from Adafruit forum

Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.
Locked
User avatar
jarchie
 
Posts: 615
Joined: Sun Jun 24, 2012 2:16 pm

Timesquare Watch and BOD-disable

Post by jarchie »

I noticed the following comment in the code for the Timesquare Watch.
// VITALLY IMPORTANT: brown-out reset (BOR) is NOT disabled, even though
// this can save many micro-Amps during power-down sleep. As the coin
// cell runs down, real brown-outs are quite likely. And when this
// happens, if BOR is disabled, the MCU will behave erratically and may
// jump to any random location...and if this leads into any bootloader
// code that erases or writes a flash page, the application -- or much
// worse, the bootloader itself -- can become corrupted, leaving no easy
// way to re-flash the watch. This is NOT the unlikely one-in-a-million
// chance you might think...actual odds seem to be about 1% -- the
// phenomenon has been observed in the wild with other projects and even
// while developing this code. So BOR is left enabled to provide a
// proper safety net. Really, DO NOT go adding BOR-disabling code,
// you'll regret it later. Just don't. Okay? Don't. Thanks.
But the ATMEGA328P datasheet clearly states the opposite.
If BOD is disabled in software, the BOD function is turned off immediately after entering the sleep mode. Upon wake-up from sleep, BOD is automatically enabled again. This ensures safe operation in case the VCC level has dropped during the sleep period.

When the BOD has been disabled, the wake-up time from sleep mode will be approximately 60 μs to ensure that
the BOD is working correctly before the MCU continues executing code.
Is the datasheet wrong?

User avatar
pburgess
 
Posts: 4161
Joined: Sun Oct 26, 2008 2:29 am

Re: Timesquare Watch and BOD-disable

Post by pburgess »

It's either incorrect or maybe just poorly worded. There isn't a sufficient explanation of what occurs if a brownout occurs during sleep with BOD disabled...sure, it's re-enabled on wake-up before resuming code, but what if the program counter is already clobbered by that point due to the brownout? Or the contents in RAM?

Empirical evidence points to disabling BOD being a bad idea when a bootloader is involved (as is the case with the Timesquare and most any other Arduino-like derivative). We've clobbered a number of chips this way and they needed to be re-flashed. Leave BOD enabled, no such problem.

User avatar
jarchie
 
Posts: 615
Joined: Sun Jun 24, 2012 2:16 pm

Re: Timesquare Watch and BOD-disable

Post by jarchie »

Thank you for the quick reply.

I still believe the ATMEGA328P datasheet contradicts the behavior you describe, but I'm not suggesting that the behavior you describe doesn't happen. In the bootloaders section, the datasheet claims
Flash corruption can easily be avoided by following these design recommendations (one is sufficient): [...] 3. Keep the AVR core in Power-down sleep mode during periods of low VCC.
The Atmel "picoPower Basics" guide (AVR4013) also claims
Brown out detection is not required in the power-save sleep mode because no data can be corrupted.
So in three separate places, the Atmel literature seems to claim that disabling BOD in software is safe. To me, that indicates that the ATMEGA328P is designed to withstand brownouts during sleep. Perhaps naively, I imagine a system where the voltage required to maintain a sleep state is higher than the voltage at which data corruption occurs. If that were true, the chip would effectively be "off" by the time corruption of data (program counter, RAM, etc.) could occur, and at that point, corruption wouldn't matter.

Of course I still might be interpreting the datasheet incorrectly, but I'm starting to wonder if the behavior you describe isn't a hardware bug in the ATMEGA328P. If you're confident that the failures were due to disabling BOD in software, it might be worth reporting to Atmel.

User avatar
jarchie
 
Posts: 615
Joined: Sun Jun 24, 2012 2:16 pm

Re: Timesquare Watch and BOD-disable

Post by jarchie »

I apologize for pressing this issue, but disabling BOD during sleep is among my favorite power-saving techniques. If possible, I'd like to replicate the failures you describe in the hopes of finding a workaround. If I'm very lucky, maybe I could even convince Atmel to address the issue in future hardware revisions.

Today I stumbled across the Atmel picoPower(TM) technology training module which describes why disabling BOD during sleep is supposed to be safe:
While in sleep mode, the only critical parameters to handle are the RAM and register contents. On AVR microcontrollers these contents are valid until ~0.3V Vcc, while the AVR's Power-on Reset (POR) triggers at ~1.0V. If a power supply voltage drop should occur while in sleep mode with the BOD disabled, the SRAM and register contents will be valid until a POR occurs.
I certainly appreciate the information you've already provided, but would like more details. Are you confident you observed sleep-BOD-disable-related failures or did sleep-BOD-disable-related failures just seem like the most likely cause? Did you need to install a new Arduino bootloader or was it only the Arduino sketch that needed to be reloaded? Is there anything you can remember that could help me replicate the issue?

User avatar
jarchie
 
Posts: 615
Joined: Sun Jun 24, 2012 2:16 pm

Re: Timesquare Watch and BOD-disable

Post by jarchie »

...bump...

If there's nothing else that pburgess (PaintYourDragon) or Adafruit can share, then I completely understand, but I would really appreciate a reply of some sort.

User avatar
pburgess
 
Posts: 4161
Joined: Sun Oct 26, 2008 2:29 am

Re: Timesquare Watch and BOD-disable

Post by pburgess »

Hi there,

Sorry, it's been a combination of super busy here and super lengthy to explain all the reasons for the BOD decision. It's been a while, so I don't even remember all the reasons...just that I struggled with it personally for weeks before bringing it up with Limor, who was like, "Oh yeah, that. Ran into that with the x0xb0x. Don't disable BOD when you have a bootloader, bad mojo happens." So there's apparently a long precedent with this, from a much more competent engineering authority than myself.

With the watch, I think the problem would randomly manifest when changing batteries. So it wasn't just a slightly-lowish-voltage thing, but a drastic drop (maybe just power from the decoupling cap). It might have been a result of:
1) Me doing something boneheaded.
2) Something not correctly described in the datasheet.
3) Some design flaw in the chip.

Of these, #1 seems the most likely, even though I was modeling the power-down functionality on various examples from Atmel documents and existing libraries that others were using without complaint (though maybe without a bootloader). So:
A) Maybe it can be fixed, or
B) Maybe it can't.

But given that:
A) Weeks were already sunk into this one problem alone, making a late product later,
B) More weeks would need to be sunk into a fix, if one even exists, making it later still,
C) Creating a repeatable test case was pretty much impossible, as some of the conditions appeared random, so asking for support could be problematic,
D) We're talking numbers in maybe the hundreds over the product lifetime, not like a major vendor shipping tens of thousands of something in a month, so I wouldn't exactly expect Atmel to be doing gymnastics for us,
E) Numbers in maybe the hundreds over the product lifetime again, $30 kit, engineer salaries, finite hours in day,
F) Even with the non-optimal BOD setting, we still estimated about 9 months sleep time,
G) The LED matrix will clobber the battery long before that,
H) If there were some revision of the silicon that addressed this (if it's really a chip problem at all and not software boneheadery), we'd have firmware that would only run on a specific revision of a chip that even as it stands has run into supply problems now and then,
I) Maybe it really is boneheadery and a solution will present itself later; yay, open source, can issue new firmware at such time,
J) If it were something like a medical device there might be more motivation toward that last detail, but as it is it's simply a fun learn-to-solder kit, not a pacemaker, maybe wringing that last 0.01% shouldn't be the top priority,

Vs.:
A) Leave BOD enabled during power-down sleep. Problem does not manifest. Ship. People build kit and are pretty happy with it.

It just made more sense to drop it and move on to other, bigger issues, of which there were many. The power usage during sleep was a flea fart in comparison. Half a flea fart, in fact. Given the certain result of fixing massive power draw issues with the LED matrix vs. a time-consuming and highly uncertain flea fart hunt, it was pretty obvious where resources should be committed.

User avatar
jarchie
 
Posts: 615
Joined: Sun Jun 24, 2012 2:16 pm

Re: Timesquare Watch and BOD-disable

Post by jarchie »

That totally makes sense. And thank you so much for the detailed reply! You're awesome!

If I am able to duplicate the problem, I'll update this thread accordingly. But I probably won't have time to even start that project for at least three weeks. :-(

And again... thank you, thank you, thank you.

User avatar
pburgess
 
Posts: 4161
Joined: Sun Oct 26, 2008 2:29 am

Re: Timesquare Watch and BOD-disable

Post by pburgess »

Cool beans. I was worried it might look like we took a "couldn't be bothered" brush-off to the BOD problem...but actually we were bothered quite a bit before eventually giving in and moving on. :wink:

User avatar
jarchie
 
Posts: 615
Joined: Sun Jun 24, 2012 2:16 pm

Re: Timesquare Watch and BOD-disable

Post by jarchie »

Okay... I finally had time to look into this issue. I'm quite enamored with the TimeSquare kit, and the code is nothing short of gorgeous. Bravo!

A major selling point of Atmel's picoPower chips is that they can safely run on <1uA, but the only way to do that is disabling BOD during sleep. So if disabling BOD during sleep is unsafe, it's a huge deal, which is why I think this issue is worth looking into.

Unfortunately, I was unable to replicate the BOD-disable problem. How certain are you that the problems during testing were related to disabling BOD? Is it possible that you were disabling BOD in the fuses (definitely unsafe) rather than in the code (allegedly safe--at least according to Atmel)?

I still have the test setup described below in a corner of my closet. Is there anything I could be doing better?

EDIT: Just to be clear, I am not criticizing the TimeSquare design or code, and I agree that there is little to no benefit of lowering sleep current further in the watch. But I am trying figure out if my favorite power-saving trick--disabling BOD during sleep--is actually something I should stop doing, at least with a bootloader.


Testing method: On the TimeSquare, I changed the sleep code to disable BOD in Watch.cpp:

Code: Select all

  cli();
  uint8_t mcucr1, mcucr2;
  mcucr1 = MCUCR | _BV(BODS) | _BV(BODSE);  //turn off the brown-out detector
  mcucr2 = mcucr1 & ~_BV(BODSE);
  MCUCR = mcucr1;
  MCUCR = mcucr2;
  sei(); // Keep interrupts enabled during sleep
  sleep_cpu();
I'm not sure if it was necessary, but I also disabled the analog comparator with "ACSR = _BV(ACD);" in the Watch::begin() function. With those changes, the watch consumed less than a microamp while sleeping.

To simulate the kind of brown-outs that might happen during battery replacement, I breadboarded a DC Boarduino to control the simulation, a LM317 to provide battery-like voltage, and three n-FETs to toggle buttons and power. A 0.1 uF cap in parallel with the watch allowed voltage to sag more gradually after the power was turned off:
BOD-disable test setup
BOD-disable test setup
breadboard.jpg (53.34 KiB) Viewed 1503 times
The DC Boarduino ran the following sketch. In the loop, the code allows the watch to sleep, cuts watch power for a specified amount of time, restores power, and presses a button to wake the watch.

Code: Select all

#define SET_FET   2  // digital 2 (PD2) on timesquare 
#define PLUS_FET  3  // digital 3 (PD3) on timesquare
#define POWER_FET 4

//#define OFF_TIME 199  // max w/ multimeter
#define OFF_TIME 258  // max w/o multimeter


void setup() {
  pinMode(SET_FET,   OUTPUT);
  pinMode(PLUS_FET,  OUTPUT);
  pinMode(POWER_FET, OUTPUT);
  
  // start with power off and no buttons pressed
  digitalWrite(SET_FET,   LOW);
  digitalWrite(PLUS_FET,  LOW);
  digitalWrite(POWER_FET, LOW);
  delay(1000);
  
  // power on; wait for startup
  digitalWrite(POWER_FET, HIGH);
  delay(2000);
  
  // push button for 0.1 seconds
  digitalWrite(SET_FET, HIGH);
  delay(100);
  digitalWrite(SET_FET, LOW);
  delay(100);
  
  // exit set-time mode
  digitalWrite(SET_FET,  HIGH);
  digitalWrite(PLUS_FET, HIGH);
  delay(2500);
  digitalWrite(SET_FET,  LOW);
  digitalWrite(PLUS_FET, LOW);
  delay(5000);
}


void loop() {
  // briefly turn off power
  digitalWrite(POWER_FET, LOW);
  delay(OFF_TIME);
  digitalWrite(POWER_FET, HIGH);
  delay(5000);
  
  // display time
  digitalWrite(SET_FET, HIGH);
  delay(100);
  digitalWrite(SET_FET, LOW);
  delay(5000);
}
With a multimeter measuring voltage, the watch could consistently endure a 199ms power loss while sleeping, which caused voltage to sag to just under one volt. A longer power loss caused the watch to reset when power was restored. That makes sense because, according to the Atmel literature, the RAM/registers are valid to ~0.3v, but a voltage below ~1v will trigger a power-on reset when power is restored, thereby ensuring the validity of the RAM/registers when BOD is disabled during sleep.
Voltage during test
Voltage during test
measurements.jpg (30.7 KiB) Viewed 1503 times
It surprised me that my multimeter was pulling enough current to make a significant difference in the maximum length of a power-loss, but it did. So for my actual testing, I increased the duration of the power loss to 258ms.

After over 12 hours and more than 3000 simulated brown-outs, the TimeSquare watch still works perfectly. If the probability of program corruption was 1%, the probability of no corruption is 99%. That means the probability of no corruption after 3000 trials would be 0.99^3000--less than one in a trillion.

EDIT 2: My test methodology was quite specific, so I only proved that my brown-out test was not sufficient to cause program memory corruption with a frequency of 1%. It is entirely possible that a different test would cause program memory corruption with a frequency of 1%, as claimed in the comment I quoted in the initial post of this thread.

pburgess wrote:...bringing it up with Limor, who was like, "Oh yeah, that. Ran into that with the x0xb0x. Don't disable BOD when you have a bootloader, bad mojo happens." So there's apparently a long precedent with this...
I was looking at the parts list for the x0xb0x, and it uses the ATmega162, which is not a picoPower device and does not support software BOD-disable. The only way to disable BOD would have been in the fuses, and disabling BOD that way will definitely cause problems (unless lockbits are set to prevent self-programming).

Locked
Please be positive and constructive with your questions and comments.

Return to “Clock Kits (discontinued)”