Beaglebone almost-stock Memory leak?

[nullop]

I bought a new Beaglebone from adafruit this week with a wi-fi dongle.

I followed the Adafruit guide for adding wi-fi access which worked fine, it's connected to my unencrypted guest network, gets an IP address, can ping google, etc.

I made zero other changes other than the 3 opkg commands, editing the connman config file and restarted that service. I rebooted the device fresh, tested its IP connectivity, worked completely fine.

Walked away from console around 2am, and came back around 9am.. and it was no longer connected, and was mostly non-functional, appearing to have run out of memory. I've added no other utilites, no software, nothing.

I've yet to go through and start figuring out what's leaking, just curious if everyone else has seen the same behavior with or without wifi? My intention for this device would put it in a remote location untouched by humans for long periods of time inputting data to Amazon SQS.. having it lockup due to memory issues before code is even written makes it seem less than ideal.


Here was the console log when I came back, look around second 25368

root@beaglebone:~# [ 3840.100367] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
[ 3840.107374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3960.110362] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
[ 3960.117369] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3960.125632] INFO: task wpa_supplicant:176 blocked for more than 120 seconds.
[ 3960.133000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4080.140367] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
[ 4080.147370] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4080.155630] INFO: task wpa_supplicant:176 blocked for more than 120 seconds.
[ 4080.163001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4200.170373] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
[ 4200.177377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4200.185635] INFO: task wpa_supplicant:176 blocked for more than 120 seconds.
[ 4200.193001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4320.200376] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
[ 4320.207382] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4320.215642] INFO: task wpa_supplicant:176 blocked for more than 120 seconds.
[ 4320.223009] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4440.230372] INFO: task kworker/0:1:20 blocked for more than 120 seconds.
[ 4440.237375] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[25368.185123] Out of memory: Kill process 100 (node) score 3 or sacrifice child
[25368.192670] Killed process 100 (node) total-vm:52296kB, anon-rss:8328kB, file-rss:0kB
[26428.607749] Out of memory: Kill process 58 (systemd-journal) score 1 or sacrifice child
[26428.616176] Killed process 58 (systemd-journal) total-vm:4796kB, anon-rss:192kB, file-rss:232kB
[26428.871747] Out of memory: Kill process 94 (connmand) score 1 or sacrifice child
[26428.879477] Killed process 94 (connmand) total-vm:4992kB, anon-rss:1532kB, file-rss:0kB
[26433.250837] mmcblk0: error -110 transferring data, sector 546897, nr 8, cmd response 0x900, card status 0x200b00
[26436.057840] mmcblk0: error -110 transferring data, sector 546897, nr 8, cmd response 0x900, card status 0x0
[26436.068041] end_request: I/O error, dev mmcblk0, sector 546897
[26438.870449] mmcblk0: error -110 transferring data, sector 546898, nr 7, cmd response 0x900, card status 0x0
[26438.880651] end_request: I/O error, dev mmcblk0, sector 546898
[26441.683039] mmcblk0: error -110 transferring data, sector 546899, nr 6, cmd response 0x900, card status 0x0
[26441.693237] end_request: I/O error, dev mmcblk0, sector 546899
[26445.861592] mmcblk0: error -110 transferring data, sector 546900, nr 5, cmd response 0x900, card status 0x0
[26445.871802] end_request: I/O error, dev mmcblk0, sector 546900
[26448.674189] mmcblk0: error -110 transferring data, sector 546901, nr 4, cmd response 0x900, card status 0x0
[26448.684391] end_request: I/O error, dev mmcblk0, sector 546901
[26451.486778] mmcblk0: error -110 transferring data, sector 546902, nr 3, cmd response 0x900, card status 0x0
[26451.496974] end_request: I/O error, dev mmcblk0, sector 546902
[26454.299352] mmcblk0: error -110 transferring data, sector 546903, nr 2, cmd response 0x900, card status 0x0
[26454.309560] end_request: I/O error, dev mmcblk0, sector 546903
[26458.476465] mmcblk0: error -110 transferring data, sector 546904, nr 1, cmd response 0x900, card status 0x0
[26458.486670] end_request: I/O error, dev mmcblk0, sector 546904
[26462.659278] mmcblk0: error -110 transferring data, sector 546897, nr 8, cmd response 0x900, card status 0x200b00
[26465.466265] mmcblk0: error -110 transferring data, sector 546897, nr 8, cmd response 0x900, card status 0x0
[26465.476466] end_request: I/O error, dev mmcblk0, sector 546897
[26468.278856] mmcblk0: error -110 transferring data, sector 546898, nr 7, cmd response 0x900, card status 0x0
[26468.289054] end_request: I/O error, dev mmcblk0, sector 546898
[26471.091440] mmcblk0: error -110 transferring data, sector 546899, nr 6, cmd response 0x900, card status 0x0
[26471.101636] end_request: I/O error, dev mmcblk0, sector 546899
[26473.904032] mmcblk0: error -110 transferring data, sector 546900, nr 5, cmd response 0x900, card status 0x0
[26473.914228] end_request: I/O error, dev mmcblk0, sector 546900
[26476.716609] mmcblk0: error -110 transferring data, sector 546901, nr 4, cmd response 0x900, card status 0x0
[26476.726807] end_request: I/O error, dev mmcblk0, sector 546901
[26479.529192] mmcblk0: error -110 transferring data, sector 546902, nr 3, cmd response 0x900, card status 0x0
[26479.539389] end_request: I/O error, dev mmcblk0, sector 546902
[26482.341770] mmcblk0: error -110 transferring data, sector 546903, nr 2, cmd response 0x900, card status 0x0
[26482.351967] end_request: I/O error, dev mmcblk0, sector 546903
[26485.154350] mmcblk0: error -110 transferring data, sector 546904, nr 1, cmd response 0x900, card status 0x0
[26485.164551] end_request: I/O error, dev mmcblk0, sector 546904

[adafruit]

what did 'top' and other tools say?

[nullop]

what did 'top' and other tools say?

haven't had a chance to debug, it wasn't accepting shell input in the state it was left in.

Left it in that state all day, and some time during the day it ended up kernel panicking:

[29104.659936] [<c003834c>] (kthread_data+0x4/0xa) from [<c0036251>] (wq_worker_sleeping+0x9/0x54)
[29104.669024] [<c0036251>] (wq_worker_sleeping+0x9/0x54) from [<c028d583>] (__schedule+0xb3/0x380)
[29104.678194] Code: 25d4 c029 f8d0 3240 (f853) 0c04
[29104.683226] ---[ end trace dc7c775e962eb9dc ]---
[29104.688041] Fixing recursive fault but reboot is needed!
[29104.693598] Unable to handle kernel paging request at virtual address fffffffc
[29104.701133] pgd = c0004000
[29104.703953] [fffffffc] *pgd=8fffe821, *pte=00000000, *ppte=00000000
[29104.710504] Internal error: Oops: 17 [#11]
[29104.714774] Modules linked in: arc4 rtl8192cu rtl8192c_common rtlwifi g_mass_storage mac80211 cfg80211 rfkill ipv6
[29104.725614] CPU: 0 Tainted: G D (3.2.5+ #1)
[29104.731251] PC is at kthread_data+0x4/0xa
[29104.735435] LR is at wq_worker_sleeping+0x9/0x54
[29104.740253] pc : [<c003834c>] lr : [<c0036251>] psr: 000001b3
[29104.740259] sp : cfa22300 ip : 00000020 fp : c037e480
[29104.752235] r10: cfa22308 r9 : cf9df7ec r8 : c0478a88
[29104.757683] r7 : cfa22318 r6 : c0478a88 r5 : 0000000b r4 : 00000000
[29104.764491] r3 : 00000000 r2 : cf801640 r1 : 00000000 r0 : cf9df580
[29104.771300] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA Thumb Segment user
[29104.779016] Control: 50c5387d Table: 8f998019 DAC: 00000015
[29104.785009] Process F (pid: 6844998, stack limit = 0xcfa222f0)
[29104.792181] Stack: (0xcfa22300 to 0x0fbfe845)
[29104.796736] [<c003834c>] (kthread_data+0x4/0xa) from [<c0036251>] (wq_worker_sleeping+0x9/0x54)
[29104.805825] [<c0036251>] (wq_worker_sleeping+0x9/0x54) from [<c028d583>] (__schedule+0xb3/0x380)
[29104.814996] Code: 25d4 c029 f8d0 3240 (f853) 0c04
[29104.820006] Unable to handle kernel NULL pointer dereference at virtual address 000000d0
[29104.828447] pgd = c0004000
[29104.831266] [000000d0] *pgd=00000000
[29104.834996] Internal error: Oops: 17 [#12]
[29104.839263] Modules linked in: arc4 rtl8192cu rtl8192c_common rtlwifi g_mass_storage mac80211 cfg80211 rfkill ipv6
[29104.850097] CPU: 0 Tainted: G D (3.2.5+ #1)
[29104.855732] PC is at account_system_time+0xca/0x114
[29104.860835] LR is at kstat+0x20/0x80
[29104.864565] pc : [<c0026202>] lr : [<c0478f70>] psr: 000001b3
[29104.864571] sp : cfa21fa8 ip : 0000001a fp : 00001a78
[29104.876548] r10: 7e702c40 r9 : 00000000 r8 : c047ab90
[29104.881998] r7 : cfa21fa8 r6 : cfa200e0 r5 : 00000000 r4 : 00000001
[29104.888809] r3 : 00000000 r2 : 00000001 r1 : 00000000 r0 : 01a20000
[29104.895620] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA Thumb Segment user
[29104.903338] Control: 50c5387d Table: 8f998019 DAC: 00000015
[29104.909333] Process (pid: 1, stack limit = 0xcfa202f0)
[29104.914781] Stack: (0xcfa21fa8 to 0x00002000)
[29104.919329] Code: f8c6 3258 f8d6 3374 (f8d3) 10d0
[29104.924327] ---[ end trace dc7c775e962eb9dd ]---
[29104.929142] Kernel panic - not syncing: Fatal exception in interrupt
[29035.885743] gadget: high-speed config #1: Linux File-Backed Storage

[nullop]

though, I may have something faulty. Fresh reboot, and instantly spitting i/o error for mmcblk0

[ 92.853688] mmcblk0: error -110 transferring data, sector 5308502, nr 43, cmd response 0x900, card status 0x0
[ 92.864100] end_request: I/O error, dev mmcblk0, sector 5308502
[ 144.925103] mmcblk0: error -110 transferring data, sector 5308496, nr 1, cmd response 0x900, card status 0x0
[ 144.935379] end_request: I/O error, dev mmcblk0, sector 5308496
[ 144.942123] Kernel panic - not syncing: Attempted to kill init!


Others have commented about faulty SD cards with their beaglebones, but the threads I found were people storing databases on them and doing tons of writes. This is a failed card after less than 24 hours of running with no apps at all. (Though i suppose whatever is running it out of memory is causing a fairly large set of logs to be written, but even the crappiest SD card should be able to handle 24 hours of writing).

Thoughts? Looks like its toast at this point. Do you have an exact image of what you're shipping on these cards that I can dd onto another card?

[nullop]

Re-formatted the card and wrote this image to it http://downloads.angstrom-distribution. ... .09.img.xz
(Would have used a different card, but don't have any micro-SD cards I could spare for this project at the moment)

boots again.. ran through opkg updates, it's an entirely different type of unstbale now.

just table completing 'dmesg' takes 10-15 seconds.. sometimes when it boots it has wlan0, sometimes it doesn't. (using 5V power cord from adafruit). Didn't have those issues yesterday.

iwconfig hangs when you run it, etc. Assuming this is likely related to the SD card still.

[adafruit]

check if you have a nice clean source of power - this can be a tricky thing especially with the wifi power hogging

[nullop]

check if you have a nice clean source of power - this can be a tricky thing especially with the wifi power hogging

Currently using the 5V adapter from Adafruit, and it's running on my bench which uses a power conditioner, it's a clean/steady 119.

I picked up an Amazon Basics class 10 4gig card and a sandisk class 6 4gig card, will try both of them tonight and see if replacing the SD makes it happier.

[adafruit]

might also be good to post up on the beaglebone forum - they're in charge of wrapping the distro and also know chip internals!

[nullop]

might also be good to post up on the beaglebone forum - they're in charge of wrapping the distro and also know chip internals!

May have to do that. Now with a new SD card it seemed a little healthier, but wireless is just miserable.

3/4th of the time when it boots it never powers on the card (5V is hooked up and stable, im rebooting by using the reset button next to the ethernet interface.. as the unit is typically locked up and 'reboot' doesnt work)

when it does come on, it connects as it should.. but after some amount of time (unclear how long, an hour maybe) it disconnects, indicates its still connected via iwconfig.. (usually, sometimes iwconfig locks up itself.. ive tried to catch this in gdb but seem to fail).

Not sure, the device seems pretty stable when using ethernet, but wireless is is not a great/stable experience. Certainly not something one could setup, and embed this device somewhere and expect it to work reliably.

[nullop]

Looks like memory leak prevails no matter what, let it run all night, pinging google.. (which perhaps keeping network traffic is keeping it connected? The infrastructure isn't timing it out. will have to test more on that).

Anyway, still died out of memory. Will have to dig more into that piece too. Once it hits the memory issue, console no longer responds, you just see the errors.

64 bytes from sea09s01-in-f0.1e100.net (173.194.33.0): icmp_req=33325 ttl=54 time=27.6 ms
64 bytes from sea09s01-in-f0.1e100.net (173.194.33.0): icmp_req=33326 ttl=54 time=29.4 ms
[33617.926635] Out of memory: Kill process 58 (systemd-journal) score 1 or sacrifice child
[33617.935028] Killed process 58 (systemd-journal) total-vm:45544kB, anon-rss:208kB, file-rss:16kB
[33618.053619] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.061065] Killed process 312 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.277648] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.285156] Killed process 314 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.603637] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.611083] Killed process 317 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.643676] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.651123] Killed process 321 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.683197] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.690643] Killed process 324 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.730651] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.738159] Killed process 327 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.783416] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.790863] Killed process 330 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.813293] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.820739] Killed process 333 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.843261] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.850738] Killed process 336 (crond) total-vm:3500kB, anon-rss:576kB, file-rss:0kB
[33618.872741] Out of memory: Kill process 93 (crond) score 1 or sacrifice child
[33618.880218] Killed process 93 (crond) total-vm:2968kB, anon-rss:516kB, file-rss:0kB
[34004.123321] Out of memory: Kill process 94 (connmand) score 1 or sacrifice child
[34004.131042] Killed process 94 (connmand) total-vm:3992kB, anon-rss:364kB, file-rss:4kB
[34004.164001] Out of memory: Kill process 95 (avahi-daemon) score 1 or sacrifice child
[34004.172119] Killed process 136 (avahi-daemon) total-vm:3084kB, anon-rss:176kB, file-rss:0kB
[34004.203247] Out of memory: Kill process 95 (avahi-daemon) score 1 or sacrifice child
[34004.211334] Killed process 95 (avahi-daemon) total-vm:3084kB, anon-rss:236kB, file-rss:0kB
[36158.155059] ICMPv6 ND: ndisc_build_skb() failed to allocate an skb, err=-105.
[36158.354980] ICMPv6 ND: ndisc_build_skb() failed to allocate an skb, err=-105.
[36159.554901] ICMPv6 ND: ndisc_build_skb() failed to allocate an skb, err=-105.

[nullop]

Also as a note, running off of ethernet without the wireless dongle in, don't run into the 'out of memory' issues. Just sitting idle, seems to be completely stable. past 20 hours haven't had a single console output error. Normally it'd be locked up by now.