言午月月鸟 - [追查] 一个空格引发的事故

### 问题

线上一个服务出现异常问题，报错：redis连接不上。

经追查，是配置文件中redis host配置处多了一个空格，导致 `redis->connect` 失败；修改配置重启，系统恢复，止损完毕。

但这个配置文件的修改应该是很久以前（早在去年11月）就发生的，应用只要发生过重启应该会读取。为何之前都没有问题，刚刚才触发？

### 处理

下午2点多，告警群开始告“`redis连接失败`”。同时系统xx功能不可用。

查看日志，显示：（已脱敏）

```sh
# 框架日志
PHP Warning:  Redis::connect(): connect() failed: php_network_getaddresses: getaddrinfo failed: Name or service not known in /path/to/log/RedisCli.php on line 112

# 应用日志 
20210804-142705-712|DEBUG|xxxxx|redis_connect ： host:10.10.1.111  port:6379
```

测试是否网络问题：

在机器上 `telnet 10.10.2.4 6379`，可以连通。

测试是否PHP类库问题：

在机器上tmp目录写一个临时代码 `/tmp/t.php`，测试可以正常操作redis。

```php
<?php

try {
  $redis = new Redis();
  $redis->connect('10.10.2.4', 6379);
  $redis->auth('crs-g0dg7okv:vZ3RScfBUI');
  $tmpKey = 'tmp:test';
  echo 'test: ', $redis->incr($tmpKey), PHP_EOL;
} catch (Exception $e) {
  echo 'Exception: ', $e->getMessage(), PHP_EOL;
}
```

查看配置文件：`/path/to/config.xml`

发现其中配置，REDIS->Recycle->Host后面多了一个空格，怀疑与此有关。

```XML
<REDIS>
    <Host>10.10.1.111 </Host>
    <Port>6379</Port>
    <Auth>123456</Auth>
</REDIS>
```

修改上面 `/tmp/t.php` 脚本，增加空格：`$redis->connect('10.10.2.4 ', 6379);` 复现相同报错。

修改配置，去掉空格，重启应用。系统恢复。

### 新的疑问

但进一步追查原因又陷入困惑。

从日志看，08-03及之前时间，配置文件已经是带了空格的情况，但系统并没有问题。

### 初步判断

之前运行环境，可以兼容 php 的 redis client 带空格的Host连接。

但到08-04 时，发生了某个（些）变化，导致不再兼容。

找运维了解到，期间执行过类库安装：

```Bash
yum -y install libjpeg-devel
echo "libjpeg-devel OK"
yum -y install curl-devel
echo "curl-devel OK"
yum install glibc-headers -y
echo "headers OK"
yum install gcc-c++ -y
echo "gcc OK"
yum install libxml2-devel -y
echo "libxml2 OK"
yum -y install libicu-devel
echo "libicu OK"
```

大概率是这中间某个依赖导致了系统环境某些变化。

### 追查redis扩展

线上机器不方便调试。经过测试上面的`/tmp/t.php` 脚本在开发机可以重现对比案例：

dev-apps01  带空格，执行失败； 不带空格，执行成功；

dev-apps03  带空格，执行成功； 不带空格，执行成功；

通过`strace php /tmp/t.php`比较（仅关键部分），可以看到：

dev-apps01带空格的时候，系统是当做域名去解析的，不带空格则直接用 ip 连接； 而 dev-apps03带空格时也是直接用ip连接：

```Bash
# dev-apps01 带空格
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1746, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f997702e000
read(3, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1746
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f997702e000, 4096)            = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=9, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f997702e000
read(3, "multi on\n", 4096)             = 9
read(3, "", 4096)                       = 0
close(3)                                = 0

# dev-apps01 不带空格
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
close(3)                                = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
connect(3, {sa_family=AF_INET, sin_port=htons(6379), sin_addr=inet_addr("10.10.2.4")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLIN|POLLOUT|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=3, revents=POLLOUT}])
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
fcntl(3, F_SETFL, O_RDWR)               = 0

# dev-apps03 带空格
socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
close(3)                                = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
connect(3, {sa_family=AF_INET, sin_port=htons(6379), sin_addr=inet_addr("10.10.2.4")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLIN|POLLOUT|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=3, revents=POLLOUT}])
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
fcntl(3, F_SETFL, O_RDWR)               = 0
```

可以看到，dev-apps03 在网络连接前，是已经去掉了空格的正确IP地址。

这应该是在 Redis::connect 方法执行中做的。

但是比较两台机器的 redis扩展，版本居然是一毛一样的。。。

```Bash
# dev-apps01
[root@dev-apps01 tmp]# php --ri redis

redis

Redis Support => enabled
Redis Version => 3.1.6
Available serializers => php, igbinary

# dev-apps03
[root@dev-apps03 tmp]# php --ri redis

redis

Redis Support => enabled
Redis Version => 3.1.6
Available serializers => php, igbinary

```

拿 `redis`扩展对应版本源码追踪，可以看到内部并没有做`host`参数的特殊处理，关键的连接建立还是依赖PHP内置的 `php_stream_xport_create `

```Bash
# redis.c
redis->sock = redis_sock_create(host, host_len, port, timeout, read_timeout, persistent,
    persistent_id, retry_interval, 0);

if (redis_sock_server_open(redis->sock TSRMLS_CC) < 0) {
    redis_free_socket(redis->sock);
    redis->sock = NULL;
    return FAILURE;
}

# libaray.c  
redis_sock->stream = php_stream_xport_create(host, host_len,
    0, STREAM_XPORT_CLIENT | STREAM_XPORT_CONNECT,
    persistent_id, tv_ptr, NULL, NULL, &err);

if (persistent_id) {
    efree(persistent_id);
}

if (!redis_sock->stream) {
    return -1;
}
```

PS：此处用 gdb 确认，传给PHP内置函数的参数仍然是带空格的：

```Bash
1457        redis_sock->stream = php_stream_xport_create(host, host_len,
(gdb) n
1461        if (persistent_id) {
(gdb) p host
$7 = "10.10.2.4 :6379\000\004\000\000\000\000\000\000\000`\347\356\364\377\177\000\000x\000\000\000\000\000\000\000\250\060K\366\377\177", '\000' <repeats 18 times>, "\060\060\241\363\377\177\000\000\000a\264\363\377\177\000\000P=K\366\377\177\000\000\036\235\336\367\377\177\000\000\001\000\000\000\000\200\377\377", '\000' <repeats 17 times>, ".\025\366\377\177\000\000\000\254\377\377\377\177\000\000`\347\356\364\377\177\000\000\005\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\060D\356UUU\000\000\350\254\377\377\377\177\000\000\340\250\377\377\377\177\000\000\350\250\377\377\377\177\000\000U\000\000\000\000\000\000\000"...
(gdb) p host_len
$8 = 15
```

PPS：根据上面调试，再次增加一个测试比较，确认了对“带空格的IP地址”的解析是PHP内部函数支持的。

```Bash
# dev-apps01 机器
[root@dev-apps01 php-7.1.33]# php -r "echo file_get_contents('http://127.0.0.1:80/'), PHP_EOL;"

Warning: file_get_contents(http://127.0.0.1:80/): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found
 in Command line code on line 1

[root@dev-apps01 php-7.1.33]# php -r "echo file_get_contents('http://127.0.0.1 :80/'), PHP_EOL;"

Warning: file_get_contents(): php_network_getaddresses: getaddrinfo failed: Name or service not known in Command line code on line 1

Warning: file_get_contents(http://127.0.0.1 :80/): failed to open stream: php_network_getaddresses: getaddrinfo failed: Name or service not known in Command line code on line 1

# dev-apps03 机器
[root@dev-apps03 tmp]# php -r "echo file_get_contents('http://127.0.0.1:80/'), PHP_EOL;"
PHP Warning:  file_get_contents(http://127.0.0.1:80/): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found
 in Command line code on line 1

[root@dev-apps03 tmp]# php -r "echo file_get_contents('http://127.0.0.1 :80/'), PHP_EOL;"
PHP Warning:  file_get_contents(http://127.0.0.1 :80/): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found
 in Command line code on line 1

```

### 追查PHP内置函数

本想继续往PHP内置函数`php_stream_xport_create`追下去，但看到上面的提示灵机一动，不如直接拿 `php_network_getaddresses`来追一下试试。

再次跑 `/tmp/t.php` 脚本，用 `gdb`调试，下断点 `php_network_getaddresses`，果然两台机器表现不一样。

在 `getaddrinfo`处，一个识别出了地址一个没有：

```Bash
# dev-apps01
Breakpoint 1, php_network_getaddresses (
    host=host@entry=0x7ffff3aee400 "10.10.2.4 ", socktype=socktype@entry=1,
    sal=sal@entry=0x7fffffffa378, error_string=error_string@entry=0x7fffffffa6d8)
    at /usr/src/debug/php-7.1.33/main/network.c:172
# 走异常分支
212             hints.ai_family = ipv6_borked ? AF_INET : AF_UNSPEC;
(gdb)
215             if ((n = getaddrinfo(host, NULL, &hints, &res))) {
(gdb)
216                     if (error_string) {

# dev-apps03
Breakpoint 1, php_network_getaddresses (
    host=host@entry=0x7ffff3aee400 "10.10.2.4 ", socktype=socktype@entry=1,
    sal=sal@entry=0x7fffffffa308, error_string=error_string@entry=0x7fffffffa668)
    at /usr/src/debug/php-7.1.33/main/network.c:172
# 走正常分支
212             hints.ai_family = ipv6_borked ? AF_INET : AF_UNSPEC;
(gdb)
215             if ((n = getaddrinfo(host, NULL, &hints, &res))) {
(gdb)
223             } else if (res == NULL) {
(gdb)
234             for (n = 1; (sai = sai->ai_next) != NULL; n++)

# 尝试 s 进入getaddrinfo，发现进入。。。不了。。。（未安装 glibc debuginfo；后面会说解决方法）

```

`getaddrinfo` 是glibc的函数。结合之前信息，很明显是 glibc 升级后，`getaddrinfo`表现不同了。

### 追查glibc函数

为了方便调试，写一个简单的 C 单独调试该函数：

```Bash
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <string.h>
int main(int argc, char **argv)
{
    if (argc != 2) {
        fprintf(stderr, "Usage: %s hostname\n", argv[1]);
        exit(1);
    }

struct addrinfo *answer, hint, *curr;
    char ipstr[16];
    bzero(&hint, sizeof(hint));
    hint.ai_family = AF_INET;
    hint.ai_socktype = SOCK_STREAM;

int ret = getaddrinfo(argv[1], NULL, &hint, &answer);
    if (ret != 0) {
        fprintf(stderr, "getaddrinfo error: %s\n", gai_strerror(ret));
        exit(1);
    }

for (curr = answer; curr != NULL; curr = curr->ai_next) {
        inet_ntop(AF_INET, &(((struct sockaddr_in *)(curr->ai_addr))->sin_addr), ipstr, 16);
        printf("IP: [%s]\n", ipstr);
    }

freeaddrinfo(answer);
    exit(0);
}

```

运行测试：

```Bash
# apps01
[root@dev-apps01 tmp]# ./addr "10.10.2.4"
IP: [10.10.2.4]
[root@dev-apps01 tmp]# ./addr "10.10.2.4 "
getaddrinfo error: Name or service not known

# apps03
[root@dev-apps03 tmp]# ./addr "10.10.2.4"
IP: [10.10.2.4]
[root@dev-apps03 tmp]# ./addr "10.10.2.4 "
IP: [10.10.2.4]

```

但是，诡异的事情又来了。

这两台机器的 内核 和 glibc版本都一致：

```Bash
[root@dev-apps01 tmp]# uname -a
Linux dev-apps01 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@dev-apps01 tmp]# ldd  --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[root@dev-apps03 tmp]# uname -a
Linux dev-apps03 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@dev-apps03 tmp]# ldd  --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

```

先试着 gdb 追踪下 dev-apps03的 `getaddrinfo` 方法吧。看看能不能找到，它在什么地方做了空格的兼容处理。

这里有个小插曲：

在 dev-apps03机器上使用 `debuginfo-install glibc-2.17-260.el7.x86_64` 安装，提示 `No debuginfo packages available to install` 。

直接上仓库 （[http://debuginfo.centos.org/7/x86_64/](http://debuginfo.centos.org/7/x86_64/)）搜索。发现：仓库里 glibc-2.17有很多个版本！（不写C不知道）

详细查看一下具体release版本，一个是 `324.el7_9`，一个是`260.el7`：

```Bash
# dev-apps01
[root@dev-apps01 devenv]# ls -l /usr/lib64/libc.so.6
lrwxrwxrwx 1 root root 12 Jul  6 16:18 /usr/lib64/libc.so.6 -> libc-2.17.so
[root@dev-apps01 devenv]# rpm -qi glibc
Name        : glibc
Version     : 2.17
Release     : 324.el7_9
Architecture: x86_64
Install Date: Tue 06 Jul 2021 04:18:12 PM CST
Group       : System Environment/Libraries
Size        : 14100682
License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
Signature   : RSA/SHA256, Thu 29 Apr 2021 11:03:27 PM CST, Key ID 24c6a8a7f4a80eb5
Source RPM  : glibc-2.17-324.el7_9.src.rpm
Build Date  : Wed 28 Apr 2021 09:58:54 PM CST
Build Host  : x86-01.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS
URL         : http://www.gnu.org/software/glibc/
Summary     : The GNU libc libraries

# dev-apps03
[root@dev-apps03 devenv]# ls -l /usr/lib64/libc.so.6
lrwxrwxrwx 1 root root 12 Jan 21  2019 /usr/lib64/libc.so.6 -> libc-2.17.so
[root@dev-apps03 devenv]# rpm -qi glibc
Name        : glibc
Version     : 2.17
Release     : 260.el7
Architecture: x86_64
Install Date: Mon 21 Jan 2019 05:24:55 PM CST
Group       : System Environment/Libraries
Size        : 14084080
License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
Signature   : RSA/SHA256, Mon 12 Nov 2018 10:26:53 PM CST, Key ID 24c6a8a7f4a80eb5
Source RPM  : glibc-2.17-260.el7.src.rpm
Build Date  : Tue 30 Oct 2018 04:21:01 PM CST
Build Host  : x86-01.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS
URL         : http://www.gnu.org/software/glibc/
Summary     : The GNU libc libraries

```

原因大概率就是这个版本差异了。

下载并装好对应glibc调试库，开始调试C程序，执行差异在 `gaih_inet `返回：

```Bash
# dev-apps01  
Breakpoint 1, __GI_getaddrinfo (name=0x7fffffffe84f "10.10.2.4 ", service=0x0,
    hints=0x7fffffffe4c0, pai=0x7fffffffe4f8) at ../sysdeps/posix/getaddrinfo.c:2208
2304          last_i = gaih_inet (name, pservice, hints, end, &naddrs, &tmpbuf);
(gdb) p last_i
$21 = 0
(gdb) p name
$22 = 0x7fffffffe84f "10.10.2.4 "
(gdb) p pservice
$23 = (struct gaih_service *) 0x0
(gdb) p hints
$24 = (const struct addrinfo *) 0x7fffffffe4c0
(gdb) p *$24
$25 = {ai_flags = 0, ai_family = 2, ai_socktype = 1, ai_protocol = 0, ai_addrlen = 0,
  ai_addr = 0x0, ai_canonname = 0x0, ai_next = 0x0}
(gdb) p end
$26 = (struct addrinfo **) 0x7fffffffdfd0
(gdb) p *$26
$27 = (struct addrinfo *) 0x0
(gdb) p naddrs
$28 = 0
(gdb) p tmpbuf
$29 = {data = 0x7fffffffe060, length = 1024, __space = '\000' <repeats 632 times>...}

# dev-apps03
Breakpoint 1, __GI_getaddrinfo (name=0x7fffffffe7ee "10.10.2.4 ", service=0x0, hints=0x7fffffffe460, pai=0x7fffffffe498) at ../sysdeps/posix/getaddrinfo.c:2208

2304          last_i = gaih_inet (name, pservice, hints, end, &naddrs, &tmpbuf);
(gdb) p last_i
$45 = 0
(gdb) p name
$46 = 0x7fffffffe7ee "10.10.2.4 "
(gdb) p pservice
$47 = <optimized out>
(gdb) p hints
$48 = (const struct addrinfo *) 0x7fffffffe460
(gdb) p *$48
$49 = {ai_flags = 0, ai_family = 2, ai_socktype = 1, ai_protocol = 0, ai_addrlen = 0, ai_addr = 0x0, ai_canonname = 0x0, ai_next = 0x0}
(gdb) p end
$50 = (struct addrinfo **) 0x7fffffffdf70
(gdb) p *$50
$51 = (struct addrinfo *) 0x602010
(gdb) p *$51
$52 = {ai_flags = 0, ai_family = 2, ai_socktype = 1, ai_protocol = 6, ai_addrlen = 16, ai_addr = 0x602040, ai_canonname = 0x0, ai_next = 0x0}
(gdb) p naddrs
$53 = 1
(gdb) p tmpbuf
$54 = {data = 0x7fffffffe000, length = 1024,
  __space = "\230\357\241\367\377\177\000\000 \003@\000\000\000\000\000\000\000\000\000\001\000\000\000P\020`\000\000\000\000\000\200\006@\000\000\000\000\000\220\345\377\377\377\177", '\000' <repeats 18 times>, "\260\344\377\377\377\177\000\000\036\235\336\367\377\177\000\000\001\000\000\000\377\177", '\000' <repeats 18 times>, "\020l\241\367\377\177\000\000 \344\377\377\377\177\000\000\352\031\337\367\377\177\000\000\356\347\377\377\377\177\000\000\230\344\377\377\377\177\000\000`\344\377\377\377\177\000\000\000\000\000\000\000\000\000\000\356\347\377\377\377\177\000\000\200^\335\367\377\177\000\000\000\000\000\000\000\000\000\000\250\344\377\367\377\177\000\000\360\340\377\377\377\177\000\000"...}

# 这里有一个地方要注意，默认打印的 ai_addr 类型不对。可以手动调整
(gdb) p $52->ai_addr
$55 = (struct sockaddr *) 0x602040
(gdb) p *$55
$56 = {sa_family = 2,
  sa_data = "\000\000\n\n\002\004\000\000\000\000\000\000\000"}
(gdb) p (struct sockaddr_in *) 0x602040
$57 = (struct sockaddr_in *) 0x602040
(gdb) p *$57
$58 = {sin_family = 2, sin_port = 0, sin_addr = {s_addr = 67242506},
  sin_zero = "\000\000\000\000\000\000\000"}
# 拿 s_addr = 67242506 整形转换为IP，即可得到 4.2.10.10
```

进一步追 `gaih_inet`方法，发现一处调用方法上有差异：

```Bash
# dev-apps01
(gdb)
482           if (req->ai_flags & AI_IDN)
(gdb)
511           if (__inet_aton_exact (name, (struct in_addr *) at->addr) != 0)
(gdb)
532           else if (at->family == AF_UNSPEC)
(gdb) p name
$4 = 0x7fffffffe84f "10.10.2.4 "
(gdb) p at->addr
$5 = {0, 0, 0, 0}

# dev-apps03
482           if (req->ai_flags & AI_IDN)
(gdb)
511           if (__inet_aton (name, (struct in_addr *) at->addr) != 0)
(gdb)
513               if (req->ai_family == AF_UNSPEC || req->ai_family == AF_INET)
(gdb) p name
$5 = 0x7fffffffe7ef "10.10.2.4 "
(gdb) p at->addr
$6 = {67242506, 0, 0, 0}

```

由 `__inet_aton`改为了 `__inet_aton_exact`，看起来是约束更严谨了。

### 结案

直接查看源码对比：`/usr/src/debug/glibc-2.17-c758a686/resolv/inet_addr.c`

```Bash
# dev-apps01 ( Release : 324.el7_9 )
/* Check whether "cp" is a valid ASCII representation of an IPv4
   Internet address and convert it to a binary address.  Returns 1 if
   the address is valid, 0 if not.  This replaces inet_addr, the
   return value from which cannot distinguish between failure and a
   local broadcast address.  Write a pointer to the first
   non-converted character to *endp.  */
static int
inet_aton_end (const char *cp, struct in_addr *addr, const char **endp)
{
  static const in_addr_t max[4] = { 0xffffffff, 0xffffff, 0xffff, 0xff };
  in_addr_t val;
  char c;
  union iaddr
  {
    uint8_t bytes[4];
    uint32_t word;
  } res;
  uint8_t *pp = res.bytes;
  int digit;
#  // 略，同 260版本的 __inet_aton 方法，多一个 endp 参数

int
__inet_aton_exact (const char *cp, struct in_addr *addr)
{
  struct in_addr val;
  const char *endp;
  /* Check that inet_aton_end parsed the entire string.  */
  if (inet_aton_end (cp, &val, &endp) != 0 && *endp == 0)
    {
      *addr = val;
      return 1;
    }
  else
    return 0;
}

# dev-apps03 ( Release : 260.el7 )
/*
 * Check whether "cp" is a valid ascii representation
 * of an Internet address and convert to a binary address.
 * Returns 1 if the address is valid, 0 if not.
 * This replaces inet_addr, the return value from which
 * cannot distinguish between failure and a local broadcast address.
 */
int
__inet_aton(const char *cp, struct in_addr *addr)
{
        static const in_addr_t max[4] = { 0xffffffff, 0xffffff, 0xffff, 0xff };
        in_addr_t val;
        char c;
        union iaddr {
          uint8_t bytes[4];
          uint32_t word;
        } res;
        uint8_t *pp = res.bytes;
        int digit;
# 略，同 324版本的 inet_aton_end 方法，少一个 endp 参数
```

可以看到新版本增加了输入字符串结束符`endp`的检查，也就是本次空格不再兼容问题的原因。结案！

PS：检查结果已经写入 `in_addr *addr`，即已经识别为正确IP地址了。

PPS：写两个C程序测试一下两个版本IP识别的不同：

```Bash
# dev-apps01
# inet.c
#include <stdio.h>
#include <stdlib.h>
#include <netinet/in.h>
#include <string.h>
int main(int argc, char **argv)
{
    if (argc != 2) {
        fprintf(stderr, "Usage: %s ip\n", argv[1]);
        exit(1);
    }

struct in_addr addr;
    fprintf(stderr, "is ipv4: %s\n", __inet_aton_exact(argv[1], &addr)==1 ? "yes" : "no");

exit(0);
}

[root@dev-apps01 tmp]# ./inet "1"
is ipv4: yes
[root@dev-apps01 tmp]# ./inet "1.0"
is ipv4: yes
[root@dev-apps01 tmp]# ./inet "1.0.0"
is ipv4: yes
[root@dev-apps01 tmp]# ./inet "1.0.0.x"
is ipv4: no
[root@dev-apps01 tmp]# ./inet "1.0.0.0xf"
is ipv4: yes
[root@dev-apps01 tmp]# ./inet "1.0.0.1 "
is ipv4: no
[root@dev-apps01 tmp]# ./inet "1.0.0.1 x"
is ipv4: no

# dev-apps03
# inet.c
#include <stdio.h>
#include <stdlib.h>
#include <netinet/in.h>
#include <string.h>
int main(int argc, char **argv)
{
    if (argc != 2) {
        fprintf(stderr, "Usage: %s ip\n", argv[1]);
        exit(1);
    }

struct in_addr addr;
    fprintf(stderr, "is ipv4: %s\n", inet_aton(argv[1], &addr)==1 ? "yes" : "no");

exit(0);
}

[root@dev-apps03 tmp]# ./inet "1"
is ipv4: yes
[root@dev-apps03 tmp]# ./inet "1.0"
is ipv4: yes
[root@dev-apps03 tmp]# ./inet "1.0.0"
is ipv4: yes
[root@dev-apps03 tmp]# ./inet "1.0.0.1"
is ipv4: yes
[root@dev-apps03 tmp]# ./inet "1.0.0.x"
is ipv4: no
[root@dev-apps01 tmp]# ./inet "1.0.0.0xf"
is ipv4: yes
[root@dev-apps03 tmp]# ./inet "1.0.0.1 "
is ipv4: yes
[root@dev-apps03 tmp]# ./inet "1.0.0.1 x"
is ipv4: yes

```