criu can't C/R ia32 processes on AMD

[avagin]

========================== Run zdtm/static/cwd02 in h ==========================
Start test
./cwd02 --pidfile=cwd02.pid --outfile=cwd02.out --dirname=cwd02.test
Run criu dump
=[log]=> dump/zdtm/static/cwd02/48/1/dump.log
------------------------ grep Error ------------------------
(00.058257) 48 was trapped
(00.058289) 48 (native) is going to execute the syscall 3, required is 15
(00.058349) 48 was trapped
(00.058375) `- Expecting exit
(00.058443) Error (compel/src/lib/infect.c:1443): Task 48 is in unexpected state: b7f
(00.058517) Error (compel/src/lib/infect.c:1451): Task stopped with 11: Segmentation fault
(00.058544) Error (criu/cr-dump.c:1482): Can't cure (pid: 48) from parasite
(00.058949) Unlock network
(00.058998) Unfreezing tasks into 1
(00.059019) 	Unseizing 48 into 1
(00.060348) Error (criu/cr-dump.c:1819): Dumping FAILED.
------------------------ ERROR OVER ------------------------
################### Test zdtm/static/cwd02 FAIL at CRIU dump ###################
Send the 9 signal to  48
Wait for zdtm/static/cwd02(48) to die for 0.100000

https://ci.openvz.org/job/CRIU/job/CRIU-x86-32/job/criu-dev/8/consoleFull

[root@usr-LAN-dhcp-38 ~]# uname -a
Linux usr-LAN-dhcp-38.99.sw.ru 4.13.5-200.fc26.x86_64 #1 SMP Thu Oct 5 16:53:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Cc: @0x7f454c46

[0x7f454c46]

Nice! Will check it.
This one is suspicious:

(00.058289) 48 (native) is going to execute the syscall 3, required is 15

[cyrillos]

Not sure if related but still https://lists.openvz.org/pipermail/criu/2017-October/039807.html we may loose error code returned in memfd_create

[avagin]

I committed your patch, but the bug is still there.

[cyrillos]

Thanks! I'll try to dig the issue after the weekend.

[sarnold1119]

Hello,
I have the same problem on Dell Inc. PowerEdge R815/04Y8PT (AMD 6180) kernel 4.15-rc9.
Also tried on HP DL580 (INTEL Intel(R) Xeon(R) CPU E7-8867 v4 @ 2.40GHz), failed differently on ia32 and ia64:
cg: Dumping cgroups for 6453
pie: 6453: __sent ack msg: 76 76 0
(00.013521) Error (criu/proc_parse.c:2393): couldn't fmemopen cgroup buffer : Invalid argument
(00.013567) ----------------------------------------
(00.013569) Error (criu/cr-dump.c:1384): Dump core (pid: 6453) failed with -1
pie: 6453: Daemon waits for command

Any progress on this issue ?

Thanks
Scott

[sarnold1119]

Same kernel on the DL580

[sarnold1119]

size appears to be 0 on the fmemopen call

[avagin]

@sarnold1119 it is another problem, could you file a new issue?

[avagin]

@sarnold1119 could you show the content of /proc/PID/cgroup for a target process?

[sarnold1119]

Update,
Works fine on Intel. Problem was cgroups configuration.

[sarnold1119]

Andrew, I just posted a update. The problem on the DL580/Intel was the cgroups config (or lack there of). Works fine on the Intel now, still broken on the AMD. We could really use the criu with ia32 on the AMD box. Thanks Scott

…

________________________________ From: Andrew Vagin [notifications@github.com] Sent: Thursday, January 25, 2018 5:50 PM To: checkpoint-restore/criu Cc: Arnold, Scott C. (JSC-CD13)[Leidos Innovations Corporation]; Mention Subject: Re: [checkpoint-restore/criu] criu can't C/R ia32 processes on AMD (#398) @sarnold1119<https://github.com/sarnold1119> could you show the content of /proc/PID/cgroup for a target process? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#398 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AiH1Y58mMksmCZ-EPA37kfkyYGY6puUyks5tORNigaJpZM4P81Nm>.

[rst0git]

Comparison of
strace -e abbrev=none -s 1000 -f criu dump -t $(pgrep test) -D checkpoint/ -v4 -o dump.log
For the process:

#include <unistd.h>

void main()
{
        while(1)
                sleep(1);
}

compiled with gcc -m32

on AMD A10 Micro-6700T APU+AMD Radeon R6 Graphics

(00.437297) Waiting for 21798 to trap
(00.437542) Daemon 21798 exited trapping
(00.437885) Sent msg to daemon 3 0 0
(00.438091) Force no-breakpoints restore
(00.438439) 21798 was trapped
(00.438700) 21798 (native) is going to execute the syscall 45, required is 15
(00.439077) 21798 was trapped
(00.439187) `- Expecting exit
(00.439641) 21798 was trapped
(00.439990) 21798 (native) is going to execute the syscall 186, required is 15
(00.440596) 21798 was trapped
(00.440711) `- Expecting exit
(00.441063) 21798 was trapped
(00.441271) 21798 (native) is going to execute the syscall 1, required is 15
pie: 21798: __fetched msg: 3 0 0
(00.441643) 21798 was trapped
(00.441749) `- Expecting exit
(00.442092) 21798 was trapped
(00.442298) 21798 (native) is going to execute the syscall 186, required is 15
(00.442790) 21798 was trapped
(00.443000) `- Expecting exit
(00.443631) 21798 was trapped
(00.443989) 21798 (native) is going to execute the syscall 186, required is 15
(00.444627) 21798 was trapped
(00.444735) `- Expecting exit
(00.445083) 21798 was trapped
(00.445289) 21798 (native) is going to execute the syscall 1, required is 15
pie: 21798: 21798: new_sp=0xf7ee3004 ip 0xf7eed049
(00.445671) 21798 was trapped
(00.445777) `- Expecting exit
(00.446121) 21798 was trapped
(00.446328) 21798 (native) is going to execute the syscall 3, required is 15
(00.446681) 21798 was trapped
(00.446787) `- Expecting exit
(00.447134) 21798 was trapped
(00.447402) 21798 (native) is going to execute the syscall 3, required is 15
(00.447982) 21798 was trapped
(00.448160) `- Expecting exit
(00.448719) Error (compel/src/lib/infect.c:1447): Task 21798 is in unexpected state: b7f
(00.448937) Error (compel/src/lib/infect.c:1455): Task stopped with 11: Segmentation fault
(00.449048) Error (criu/cr-dump.c:1390): Can't cure (pid: 21798) from parasite

on Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz

(00.071152) Waiting for 2613 to trap
(00.071198) Daemon 2613 exited trapping
(00.071273) Sent msg to daemon 3 0 0
(00.071318) Force no-breakpoints restore
(00.071402) 2613 was trapped
(00.071446) 2613 (native) is going to execute the syscall 45, required is 15
(00.071521) 2613 was trapped
(00.071544) `- Expecting exit
(00.071618) 2613 was trapped
(00.071667) 2613 (native) is going to execute the syscall 186, required is 15
(00.071741) 2613 was trapped
(00.071763) `- Expecting exit
(00.071836) 2613 was trapped
(00.071880) 2613 (native) is going to execute the syscall 1, required is 15
pie: 2613: __fetched msg: 3 0 0
(00.071967) 2613 was trapped
(00.071991) `- Expecting exit
(00.072076) 2613 was trapped
(00.072123) 2613 (native) is going to execute the syscall 186, required is 15
(00.072202) 2613 was trapped
(00.072226) `- Expecting exit
(00.072302) 2613 was trapped
(00.072348) 2613 (native) is going to execute the syscall 186, required is 15
(00.072428) 2613 was trapped
(00.072452) `- Expecting exit
(00.072529) 2613 was trapped
(00.072574) 2613 (native) is going to execute the syscall 1, required is 15
pie: 2613: 2613: new_sp=0xf7f22004 ip 0xf7f2c049
(00.072651) 2613 was trapped
(00.072675) `- Expecting exit
(00.072751) 2613 was trapped
(00.072796) 2613 (native) is going to execute the syscall 3, required is 15
(00.072873) 2613 was trapped
(00.072897) `- Expecting exit
(00.072976) 2613 was trapped
(00.073018) 2613 (native) is going to execute the syscall 3, required is 15
(00.073094) 2613 was trapped
(00.073117) `- Expecting exit
(00.073187) 2613 was trapped
(00.073229) 2613 (compat) is going to execute the syscall 173, required is 173
(00.073299) 2613 was stopped
(00.073707)

and $ grep -R 173 /usr/include/asm shows

/usr/include/asm/unistd_x32.h:#define __NR_ioperm (__X32_SYSCALL_BIT + 173)
/usr/include/asm/unistd_32.h:#define __NR_rt_sigreturn 173
/usr/include/asm/unistd_64.h:#define __NR_ioperm 173

[rst0git]

The output of strace is:

21808 wait4(-1, [{WIFSTOPPED(s) && WSTOPSIG(s) == SIGTRAP}], __WALL, NULL) = 21798
21808 write(4, "(00.447982) 21798 was trapped\n", 30) = 30
21808 write(4, "(00.448160) `- Expecting exit\n", 30) = 30
21808 ptrace(PTRACE_SYSCALL, 21798, NULL, SIG_0) = 0
21808 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=21798, si_uid=0, si_status=SIGSEGV, si_utime=0, si_stime=0} ---
21808 wait4(-1, [{WIFSTOPPED(s) && WSTOPSIG(s) == SIGSEGV}], __WALL, NULL) = 21798
21808 write(4, "(00.448719) Error (compel/src/lib/infect.c:1447): Task 21798 is in unexpected state: b7f\n", 89) = 89
21808 futex(0x7f19a475ba00, FUTEX_WAKE_PRIVATE, 2147483647) = 0
21808 write(4, "(00.448937) Error (compel/src/lib/infect.c:1455): Task stopped with 11: Segmentation fault\n", 91) = 91
21808 write(4, "(00.449048) Error (criu/cr-dump.c:1390): Can't cure (pid: 21798) from parasite\n", 79) = 79

[0x7f454c46]

Yeah, it seems from the log that rt_sigreturn() has resulted in segmentation fault.
What I would speculate - is that the sigframe is different by some reason on ia32 AMD.