Multiple SD files? Logger shield.

For other supported Arduino products from Adafruit: Shields, accessories, etc.

Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.
User avatar
adafruit_support_bill
 
Posts: 88091
Joined: Sat Feb 07, 2009 10:11 am

Re: Multiple SD files? Logger shield.

Post by adafruit_support_bill »

The SD library uses a static 512 byte buffer for the data. It is flushed when it gets full.

364 bytes of RAM is probably OK, but getting tight. This guide has some tips on optimizing your SRAM usage: http://learn.adafruit.com/memories-of-an-arduino

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

Having reduced the code in the Interrupt function as per your advice in the thread "DS1307 RTC Giving Wrong Values After Running for a Few Days" The Free Ram is now 352.
The raingauge count now advances in single counts and the data was logged to the SD card - except that I still have to comment out the If statement mentioned above.
139 rain counts (approx 1 inch of rain) gave a log file size of 3.07kB - so 352kB gives me room for almost 16000 counts? I don't know if that arithmetic is valid! If it is I don't need to worry about flushing at short intervals?
Thank you - I am getting there!

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

J ust seen your previous reply. Thanks I will look at he link. Evidently my arithmetic is not helpful!

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

You say that the SD library uses 512 Bytes as a buffer.
If one has 2 SD files open at the same time, are two segments of 512 Bytes reserved as a buffer for each file?
If not, when one of the file functions calls for a flush, will the data awaiting transfer for both files be flushed?

User avatar
adafruit_support_bill
 
Posts: 88091
Joined: Sat Feb 07, 2009 10:11 am

Re: Multiple SD files? Logger shield.

Post by adafruit_support_bill »

My understanding is that the 512 byte buffer is static, but I haven't studied the code in detail. There are other memory allocations associated with having multiple files open. It seems each open file requires about 58 bytes: http://forum.arduino.cc/index.php?topic=158304.0;wap2

Note that fat16lib is the author of the SD library, so I'd take his word as fact.

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

Thank you for the link. Looks as though it is well beyond my ability! Here is an extract from fat16lib blog.

f
fat16lib:
You can open multiple files with both the Arduino SD.h and the Adafruit version of SD.h.

Each file requires about 58 bytes of RAM. I say about since SD.h uses malloc so the amout can vary with heap fragmentation.

SD.h is subject to memory leaks. If you reuse a file handle without closing it, you will lose about 30 bytes each time you open a file.

SdFat does not use malloc and a each file requires 31 bytes.

fat16lib:
Yes, (for File Handle,) I meant a File object. Both open for write and open for read take the same amount of memory.

Code:

File file;

// other code

file = SD.open("test.txt", FILE_WRITE); // calls malloc to allocate memory

// access file here

file.close(); // calls free to release memory
Please could you say whether you think I am going to lose a block of 30 Bytes of SRAM every time this logging code runs - or even more for eack part of the data.
This extract of code, in the loop, runs each time the interrupt is triggered, when Flag is increased. The code appears to work but has not had to log more than about 130 counts. I have no "Close" instruction. I guess I am "reusing a File Object"?
Should I change to SDfat library?

Code: Select all

 if (flag != nflag){    
    {
      TimerB = millis();
      if ((TimerB - TimerA) < 60000UL){
      test = 1;
      }
    else
  test = 0;  
  TimerA = millis();

  DateTime now = RTC.now();
  mon=now.month();
  dy=now.day();
  hr=now.hour();
  mn=now.minute();
  sec=now.second();
         
      rainfile.println(flag);
      rainfile.print(",");
      rainfile.print(mon);
      rainfile.print('/');
      rainfile.print(dy);
      rainfile.print(' ');
      rainfile.print(hr);
      rainfile.print(':');
      rainfile.print(mn);
      rainfile.print(':');
      rainfile.print(sec);
 
    }
    
    if (test == 0){
    rainfile.flush(); 
    Serial.println("flushed");
  }
   nflag = flag;
}
}

User avatar
adafruit_support_bill
 
Posts: 88091
Joined: Sat Feb 07, 2009 10:11 am

Re: Multiple SD files? Logger shield.

Post by adafruit_support_bill »

I have no "Close" instruction. I guess I am "reusing a File Object"?
But you are not re-opening it either:
you will lose about 30 bytes each time you open a file.
But if you are not able to log more than 130 counts, it may be worth giving SDfat a try. It is a lower-level library and slightly less convenient to use, but as fat16lib says, it does not use malloc, which should make it more stable on a memory-constrained platform like the Arduino.

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

Don't know what i have done but the rain log file does not open any more in Open Office calc. This has happened quite often before but I don't know why.
Opening it with wordpad gives me this rubbish.

Code: Select all

ÐÏࡱá
and many lines of "y"s with 2 dots over the top. And a comment "ROOT ENTRY".

If this is because the file has not been closed, why has it sometimes worked in the past?
Sorry to pester you!

User avatar
adafruit_support_bill
 
Posts: 88091
Joined: Sat Feb 07, 2009 10:11 am

Re: Multiple SD files? Logger shield.

Post by adafruit_support_bill »

Opening it with wordpad gives me this rubbish.
Sounds like you may be writing binary data to the file instead of strings.
If this is because the file has not been closed, why has it sometimes worked in the past?
When you write to a file, you are actually writing to a buffer. The data only gets written to the card when you call close(), flush() or when the buffer fills up. If you don't run long enough to fill the buffer, nothing will get written to the card.

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

Having installed the Stop Switch that you suggested, I added some code to flush when the switch was used before removing the card and also to print "Flush"to the monitor screen to show flush was called. I also added code at the end of the rainfile loop to stop flushing after every interrupt unless there was more than 1 minute between interrupts. This might well mean that a large number of data points would be added to the buffer before that condition was reached, in continuous rainfall. But, as you say, the buffer will flush if it gets full.

There is something screwwy with the logging code as indicated by the previously reported fact that the "Error - couldn't create rnfile" check won't work with the second SD file, even though, without the check, the 2nd SD file IS written (and has had data read from it sometimes). I am wondering if the 2 faults are connected.

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

I have found that the code to detect interrupts with intervals less than 1 minute is the cause of the failure to log. SRAM is shown to be 357. with the suspect code removed. This is part of the suspect code and Test is used to prevent flushing unless the interval is more than a minute. This appears to be working according to a check on the monitor. Perhaps the milllis is too memory-hungry. More investigating tomorrow.

Code: Select all

 TimerB = millis();
      if ((TimerB - TimerA) < 60000UL){
      test = 1;
      }
    else
  test = 0;  
  TimerA = millis();

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

OK I have the code working now with 2 open files, one for ambient data and the other for a raingauge count. The program has hung a few times, but does not always, so I have included a watchdog function.
I am puzzled with Adafruits Logger00 file name code sequence which is designed to write a new file name if the old one does not exist. I do not close either file ever. I do flush after each ambient read. But if I have a power outage or if I reload the program, new filenames are always generated. Why? All the old file names do exist on the SD card and the data can be read.

Does this not imply that something is closing the open files when power fails and when a program load is done?

In some ways this is not a problem, but perhaps it ought not to work! Please would you comment.

Here is a cut and paste of the relevant code for just the ambient readings. The loop contains just the code for the raingauge and is not included here.

Code: Select all

// for the data logging shield, we use digital pin 10 for the SD cs line
const int chipSelect = 10;

// the logging files
File logfile;
File rainfile;
void error(char *str)
{
  Serial.print("error: ");
  Serial.println(str);
  while(1);
}

void setup()
{
 // initialize the SD card
  Serial.print("InitSD");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);

  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) 
  {
    error("Cd fail");
  }
  Serial.println("cdinit.");
  // create a new file
  char filename[] = "LOGGER00.CSV";
  for (uint8_t i = 0; i < 100; i++) 
  {
    filename[6] = i/10 + '0';
    filename[7] = i%10 + '0';
    if (! SD.exists(filename)) {
      // only open a new file if it doesn't exist
      logfile = SD.open(filename, FILE_WRITE); 
      break;  // leave the loop!    
    }
  }

  if (! logfile) {
    error("No lgfile");
  }

  Serial.print("Log to ");
  Serial.println(filename);
  Serial.println();

 DateTime now;
  Alarm.timerRepeat(1800, Repeats);
}
void Repeats(){
  Serial.println("30 minute timer"); 
  /* Serial.print("SRAM=");
  Serial.println(freeRam());
  */
  byte _status;
  unsigned int H_dat, T_dat;
  float RH, T_C;
  _status = fetch_humidity_temperature(&H_dat, &T_dat);
  RH = (float) H_dat * 6.10e-3;
  T_C = (float) T_dat * 1.007e-2 - 40.0;
  // fetch the time
  //now = RTC.now(); 
  logfile.print(day(), DEC);
  logfile.print("/");
  logfile.print(month(), DEC);
  logfile.print("/");  
  logfile.print(year(), DEC);
  logfile.print(" ");
  logfile.print(hour(), DEC);
  logfile.print(":");
  logfile.print(minute(), DEC);
  logfile.print(":");
  logfile.print(second(), DEC);
  logfile.print(",");

 temperature = bmp085GetTemperature(bmp085ReadUT());
  pressure = bmp085GetPressure(bmp085ReadUP());

  Serial.print(hour(), DEC);
  Serial.print(":");
  Serial.print(minute(), DEC);
  Serial.print(":");
  Serial.println(second(), DEC);


  logfile.print(T_C, 2);
  logfile.print(",");    
  logfile.print(RH, 1);
  logfile.print(",");
  logfile.print(pressure, DEC); 
  logfile.print(RH, 1);
  logfile.println();
  logfile.flush();  
  
 
  Serial.print("Pres ");
  Serial.print(pressure, DEC);
  Serial.println(" Pa");

  lcd.setCursor(0, 0);
  lcd.print("P ");
  lcd.print(pressure, DEC);
  lcd.setCursor(0, 1);
  lcd.print("RH ");
  lcd.print(RH, 1);
  lcd.print(" ");
  lcd.print("T ");
  lcd.print(T_C, 2);

  Serial.print("RH ");
  Serial.print(RH, 1);
  Serial.print("  T ");
  Serial.println(T_C, 2);
  Serial.println();
}

User avatar
adafruit_support_bill
 
Posts: 88091
Joined: Sat Feb 07, 2009 10:11 am

Re: Multiple SD files? Logger shield.

Post by adafruit_support_bill »

designed to write a new file name if the old one does not exist.
The comment may be a bit confusing. What it does is generate file names with incrementing names (logger00, logger01, logger02 etc) until it finds one that doesn't already exist, then it creates that one. On each run (up to 99), you should get a new file name.
Does this not imply that something is closing the open files when power fails and when a program load is done?
If you have done a 'flush' or written more than one buffer-load, the data will get written to disk. If you have a power outage, you may lose the last few samples.

User avatar
spiney
 
Posts: 214
Joined: Mon Jul 09, 2012 6:35 am

Re: Multiple SD files? Logger shield.

Post by spiney »

Yes I get the re-naming of each file, IF the old one does not exist. BUT IT DOES EXIST. And a new one is opened each time there is a power outage or reload - without ever having "officially" closed the files.

User avatar
adafruit_support_bill
 
Posts: 88091
Joined: Sat Feb 07, 2009 10:11 am

Re: Multiple SD files? Logger shield.

Post by adafruit_support_bill »

And a new one is opened each time there is a power outage or reload
What is the name of the new one? It should be 1 more than the old one.

Locked
Please be positive and constructive with your questions and comments.

Return to “Other Arduino products from Adafruit”