SD shield SD.OPEN verrrry slow

[robertwrand]

On the Adafruit SD datalogging shield, I am get verrrry long times on SD.OPEN, about 330ms give or take. I was thinking this might take around 10-15ms or so, so this is a big surprise... of course hardware, compiler variations could be involved.

I'm working with a MEGA 2560, Arduino compiler 1.05, using the AVRISP mkII compiler setting. (I tried the AVR ISP as well). I have the most recent Adafruit SD lib (files dated June 11, 2014) installed in the libraries folder.

At the top of my program, I have

#include <SD.h>

I initialize the board with settings found at https://learn.adafruit.com/adafruit-dat ... d-leonardo

pinMode(53, OUTPUT);

// use Adafruit command settings as of 5/22/14 rwr
if (!SD.begin(10, 11, 12, 13))
{
Serial.println("SD.begin function did not work.");
}
else
{
Serial.println("SD.begin function worked.");
}

// INITIALIZE SD CARD
while (!card.init(SPI_HALF_SPEED, 10, 11, 12, 13)) {
Serial.println("initialization failed. Things to check:");
Serial.println("* is a card is inserted?");
Serial.println("* Is your wiring correct?");
Serial.println("* did you change the chipSelect pin to match your shield or module?");

LCD_print("Check SD Card.");
}

in the loop, I have the following, triggered on a conditional, when true called once per second.

char charfile[filename.length()+1];
filename.toCharArray(charfile, sizeof(charfile));
dataFile = SD.open(charfile, O_CREAT | O_WRITE | O_APPEND);
...
write stuff...
...
dataFile.close();

The problem is that since the SD.OPEN is blocking, during the file open time I am unable to grab serial characters I need that are coming in on a serial port at 9600 Baud. The MEGA board has a 64 char hardware buffer which is great and, for my data stream (about 50 bursts per second of 8 characters per burst or about 400 chars per second), gives me about a maximum of about a 160 msec serial buffer. Once the file ops are done I read the serial port multiple times as needed to get the characters hardware-buffered during the file open and write. But 330ms for the open is too long. I'm losing characters, about 20 percent of the serial data stream per second. I need zero serial data loss which means a file open time down in the 10-20ms range if possible.

These open times are with several hundred files on the SD card inside a directory, no files in the root except for the directory.

With only a few files in the directory, the open times are about 36ms, close times about 33 msec.

Could just be the limitations of working with SD card.

Any suggestions? How can I get the SD.OPEN time reduced?
-Note: I tried card.init with SPI_FULL_SPEED but no change. Not surprised, the sd.open function is in a different lib.

[adafruit_support_mike]

The more files you have in a directory, the more work the filesystem has to do to keep track of them.

At minimum, the filesystem code has to scan through the directory data structure to find the name-to-inode mapping that will allow it to find the correct chunk of storage. At worst, the information might be stored in a data structure called a 'balanced tree' which makes access times fast, but can get expensive when you add or remove files because it has to 're-balance' the tree.

The best solution is to limit the number of files in any given directory. If your filenames advance sequentially (/001.txt, /002.txt, /003.txt, etc) just write them to subdirectories and limit the number of files in each (/00/1.txt, /00/2.txt, ... /01/0.txt, /01/1.txt, etc). If the files have names that are more or less random, create a couple layers of nested subdirectories so a file that would be named 927353.txt gets stored as /9/2/7353.txt

[robertwrand]

Thanks Mike- this is useful information, much appreciated. It's consistent with my testing over the last day, I'm getting a small but definite increase in time required as the files are added. I can definitely add another directory layer. I'm using year for the top layer, I can add month as a second layer and then day as a third layer. That will keep file counts limited per directory. I can post the results after testing.

[robertwrand]

Mike I'm somewhat perplexed to say I'm not finding a chdir function in the Adafruit SD lib. How do you change directories with your lib?

[adafruit_support_mike]

The 'open()' method works for both files and directories.

The 'listfiles' example sketch that comes with the SD library shows how to walk a file tree recursively: https://github.com/adafruit/SD/blob/mas ... tfiles.ino

[robertwrand]

Hi Mike,

I'm still having trouble finding a chdir function in the Adafruit sd lib. The SD.h file doesn't list one. I may have misunderstood you to say the Open function would do that.. it doesn't and no surprises there. How do YOU change directories using the Adafruit library for the Adafruit Sd card? Would you have an example?

[Franklin97355]

The command SD.begin(10, 11, 12, 13) starts the SD function using pins 10 to 13 which, on an UNO are the hardware SPI pins. For the mega you might try SD.begin(53,51,50,52) which are the hardware SPI pins on a mega

[robertwrand]

No problem opening the card. I can see the card characteristics and make a directory with the SD lib. I'm needing to *change* to a directory, with something like a SD.chdir (although that does not exist). How do you change to a directory with the Adafruit SD lib?

[adafruit_support_mike]

Just 'open()' the directory name.

[robertwrand]

The open() function is not working for me. Say I create a directory 2014. Then I open() it. Then I attempt to create a subdirectory such as 06 with mkdir. However the 06 dir does not show up in 2014. It is created in the root directory. Please advise.

--UPDATE: Tried mkdir with /2014/06/ and /2014/06/28/ . Those worked. Then I write files to the deepest subdir using the full path such as /2014/06/28/file.txt. That works.