Fixpoint

I sent the following to the sales guy for our building as well as the general manager last Tuesday evening, meaning they've had nearly four business days so far without a peep. In the unlikely event I hear back with something useful I'll be sure to update or comment. At the very least, I will have learned some Spanish as well as who not to use.

Re: Tamano de paquetes en nueva conexion de fibra +Movil

Estimados Cable & Wireless Panama⁽ⁱ⁾ / Mas Movil,⁽ⁱⁱ⁾

Estoy companero de apartamento de XXX (copiado) quien pregunto a YYY sobre los problemas que observe despues de cambiar a su nueva conexion fibra optica. Debido a los detalles technicos involucrados pense que sera mejor escribir a Ustedes directamente. Por favor tengan paciencia para la explicacion, y confio en que mis frustraciones se aclararan. Abajo pueden encontrar mi version original porque estoy mas fuerte en Ingles.

Para mi propia formacion, trabajaba como ingeniero de redes corporativos, programador de software, y administrador de sistemas en los EEUU, y con los an~os⁽ⁱⁱⁱ⁾ he sido cliente de varios proveedores de servicios Internet (ISPs) de tipo residencial y comercial y de varios tecnologias incluyendo telefonica (dialup), DSL, cable y fibra optica, ademas de conexiones directas en centros de datos. Por lo tanto, aunque no conozco los detalles de su red, puedo notar y diagnosticar problemas que la mayoria de sus clientes no pueden, aunque los problemas pueden afectarlos ahora o en el futuro.

Cuando eche un vistazo a la configuracion basico del router Sercomm que Ustedes nos proveyeron,^(iv) me sorprendio al ver el uso del protocolo PPPoE (Point-to-Point Protocol over Ethernet) en vez del DHCP. Como lo entiendo yo, PPPoE era una tecnologia de transicion inventado por los operadores primeros de DSL para compatibilidad con el software antiguo telefonico; ni yo ni los amigos con quienes hable lo han visto en uso desde cerca el an~o 2000, y no puedo imaginar de verdad porque sea necesario en una red moderno. Aun asi, no me preocupe al principio.

Sin embargo, mas tarde observe dos problemas, cuales creo son causadas por el uso de esta tecnologia anacronica. He podido acomodarme a la una pero no puedo evitar la otra. La causa raiz en ambos casos es que la capa del PPPoE consuma 8 octetos (bytes) por cada marco (paquete) Ethernet para sus encabezamientos, reduciendo el tamano maximo de paquetes que se puede enviar en ambos direcciones (maximum transfer unit, MTU) desde los 1500 octetos especificados por los estandares Ethernet y con apoyo amplio por el Internet, hasta solo 1492 octetos.

Mi primero problema fue que aplicaciones TCP en una sistema Linux conectaran con exito pero luego se colgaran, fallando a enviar ninguno mas trafico. Determine que eso era causada por un fracaso del descubrimiento MTU del camino ( https://en.wikipedia.org/wiki/Path_MTU_Discovery#Problems ). Ese "descubrimiento" parece necesario porque los sistemas finales no saben directamente del embotellamiento entre el router y ISP. Creo que el fracaso era causada por un router barato D-Link^(v) que usabamos a compartir la conexion, rio abajo del Sercomm, que era corrompiendo los mensajes de control necesarios ICMP. Por lo menos, pude arreglar mis programas ambos por apagar^(vi) PMTU descubrimiento en el sistema operativa local^(vii) y tambien por quitar el D-Link. Pero hago notar que ambos compartiendo y routers baratos son muy comun en America Latina!

Para el segundo problema: estoy haciendo algun trabajo de prueba para Eulora2, un juego en linea que tiene varios an~os en el desarrollo (como sucesor a Eulora, que existia en juego activo desde 2013 en paises multiples). Con la nueva conexion +Movil, el cliente no puede enviar ni recibir ningun trafico de la red en absoluto. Eso es porque usa un protocolo a la medida basada en UDP con tamano fijo de paquetes. Naturalmente, cuando usar un tamano fijo, la eleccion correcta (y mas economica) es el maximo disponible, lo que sea por supuesto los 1500 octetos. Por lo tanto, sus paquetes ya no pasan por mi conexion, y solamente la mia. Ademas de mi lucha particular, la situacion es preocupante si senala una tendencia mas grande que va a excluir un publico mas amplio del aceso al juego.

Confirme mi hipotesis sobre tamano del paquete usando pruebas "ping" con el DF (Don't Fragment) senal activado.^(viii) Otros ISPs (ambos cable y fibra, en Panama) tienen exito en entregar cargas utiles de 1472 octetos, mientras que el maximo que pasa por +Movil es de 1464.^(ix)

Ofrezco la analogia de maletas para equipaje de mano disenado para maximizar espacio mientras pasar en los compartimentos superiores del aereolinea, y ahora el avion ultimo, mas rapido y "mas moderna" disponible ha disminuido dichos compartimentos (y tambien no es posible chequear las maletas).

Yo probe cambiar el modo del router desde PPPoE hasta DHCP pero fallo de obtener una direccion (IP address).

Otro dolor de cabeza con la situacion es que, si consiga tal conexion por mi mismo, sea mas dolor instalar mi propio router en la frontera - lo que hago como procedimiento estandar para seguridad y confiabilidad - en vez de ese Sercomm, porque no tengo el software ni la experiencia en apoyar PPPoE y no veo de verdad porque debo tener que invertir en esas cosas ahora.

Por lo tanto, mis preguntas especificas para su ingenieria de redes y/o generencia son:

1. Estoy correcto en concluir que exigen el PPPoE para conectar a su servicio?

2. Como llegaron a la decision de ir en esta direccion, cuales alternativos consideraron, y consideraron las desventajas en lo mas minimo?

3. Hay alguna alternativa disponible a mi como cliente que no limita el tamano del paquete, fuera de los limites naturales del medio subyacente?

En caso de necesitan mas aclaraciones o pruebas por mi parte, favor que no vacilen a preguntar.

Attentamente,

Jacob Welsh
CTO, JWRD Computing
http://jwrd.net/

==========

Dear Cable & Wireless Panama / Mas Movil,

I'm the roommate of XXX who asked YYY about the problems I observed on switching to his new fiber optic connection. Because of the technical details involved I thought it best to write to you directly. Please bear with me for the explanation and I trust my frustrations will become clear.

For my own background, I've worked as a corporate network engineer, software developer and system administrator in the US, and have been a customer of various business and residential ISPs and technologies over the years including dialup, DSL, cable, and fiber as well as direct connections in the datacenter environment. Thus, while I do not know the details of your network, I am able to notice and diagnose problems that most of your customers will not, even if they may be affected by them now or in the future.

When I took a look at the basic settings on the Sercomm router that you provided, I was surprised to see it configured to use PPPoE (Point-to-Point Protocol over Ethernet) rather than DHCP. As I understand it, this was a transitional technology invented by early DSL operators for compatibility with legacy dialup software; neither I nor the friends I talked to have seen it in use since around the year 2000, and I can't really imagine why it would be necessary in a modern network. Still, I didn't think much of it at first.

However I later observed two problems which I believe are caused by the use of this anachronistic technology; one I was able to work around but the other cannot be avoided. The root cause in both cases is that PPPoE consumes 8 bytes per Ethernet frame (packet) for its headers, reducing the maximum packet size that can be sent in either direction (maximum transfer unit, MTU) from the 1500 bytes specified by the Ethernet standards and widely supported across the Internet, to 1492 bytes.

My first problem was that TCP applications on a Linux system would connect successfully but then freeze, failing to send any further traffic. I determined this was caused by a failure of Path MTU Discovery ( https://en.wikipedia.org/wiki/Path_MTU_Discovery#Problems ). This "discovery" seems necessary because the end systems are unaware of the MTU bottleneck at the router. I believe the failure was caused by a cheap D-Link router we'd been using to share the connection, downstream of the Sercomm, corrupting the required ICMP error messages; at least, I could get my programs working again either by disabling PMTU discovery in my operating system or by removing the D-Link. I'll point out however that both sharing and cheap routers are very common in Latin America!

For the second problem: I'm doing some testing work for Eulora2, an online game that's been in development for several years now (as the successor to Eulora which has been around and in active play since 2013 in multiple countries). With the new +Movil connection, the client is not able to send or receive any network traffic at all. This is because it uses a custom UDP based protocol with a fixed packet size. Naturally, when using a fixed size, the correct (and most economical) choice is to use the maximum size available, which of course was 1500 bytes. Thus its packets no longer fit through my connection, and mine alone. Besides my particular struggles, it's concerning if this indicates a larger trend that will exclude a wider audience from access to the game.

I confirmed my hypothesis about packet size using "ping" tests with the DF (Don't Fragment) bit set. Other ISPs (both cable and fiber, in Panama) succeed in delivering 1472 byte payloads while the maximum that gets through on +Movil is 1464.

I'll offer the analogy of carryon suitcases being designed to maximize space while fitting in the airline's overhead compartments, and now the latest, fastest and "most modern" airplane available has shrunk those compartments, and there's no way to check the bags either.

I did try changing the router from PPPoE to DHCP mode but it failed to obtain an IP address.

A further headache with the situation is that, if I were to get such a connection for myself, it would be more of a pain to install my own router at the border - which I do as standard procedure for security and reliability - in place of that Sercomm, because I don't have the software or experience with supporting PPPoE and I really don't see the justification why I should have to invest in that now.

Thus my specific questions for your network engineering and/or management are:

1. Am I correct in concluding that you require PPPoE for connecting to your service?

2. How did you arrive at the decision to go that way, what alternatives did you consider, and did you consider the downsides at all?

3. Is there any alternative available for me as a customer that does not restrict packet size, beyond the natural limits of the underlying medium?

Please don't hesitate to ask if you need further clarification or testing on my part.

Regards,

Jacob Welsh
CTO, JWRD Computing
http://jwrd.net/

1. CWP, the successor to what was until 2003 a state-owned monopoly on telephone services, now owned in part by a Liberty Latin America. [^]
2. Styled +Movil, their brand for cellular services and, recently, a "FibraFast" Internet service of up to 1Gbps symmetrical, which has been expanding aggressively, at least within the capitol city, to (re)capture the residential and small business market from cable TV provider Cable Onda / Tigo. [^]
3. These people who call themselves Latins have an "enye" which they consider a whole separate letter in its own right, despite its being transparently just an "n" with a squiggle on top and moreover absent from the actual Latin alphabet. Consequently it's also absent from the character set of the civilized computing world. I could work around this by writing "anio" or "agno" or "anyo" but those all still look odd to me, with the first two being especially phonetically dubious. Or there's "a??o" like they usually end up doing but let us speak no more of such monsters. Usually I'm just going with the plain "n", but in the case of this word, I'd soon be wishing someone a happy anus, which while undoubtedly one component of a happy new year would be rather far afield from the intended sentiment. [^]
4. Chinese make I hadn't heard of before; one of the installation techs mentioned that he preferred the Huawei model they used to install but can't anymore for political reasons. In any event it's a nicely small and tidy white box, which at first I was appreciating for being properly separated from the fiber-to-Ethernet converting modem and thus presumably interchangeable. [^]
5. DIR-615 to be specific. I'd bought the indistinguishable DIR-610 in 2016 for $21.35, strictly for use as an access point, but roomie got taken more recently for $70. Which for all I know wasn't even a ripoff and $70 is just what it takes to get a barely-functional plastic home router these days. [^]
6. "A pagar" means "to pay" while "apagar" is to turn off. "A prender" means to turn on while "aprender" is to learn. Got those all straight? Here they learn things on and pay them off. [^]
7. For Linux, net.ipv4.ip_no_pmtu_disc = 2 in /etc/sysctl.conf. Presumably then the TCP code blithely sends up to 1500-byte packets which happen to go through by virtue of fragmentation at the router, at the hidden cost of worse behavior under congested conditions. [^]
8. How to do this is OS-dependent. For BSD-likes including Mac OS X it's ping -D. For Linux with iputils it's ping -M do. I haven't found a way yet for BusyBox. [^]
9. Onto this payload ICMP adds 8 header bytes and IP another 20, reaching the 1492 mentioned earlier as maximum payload to the Ethernet frame. [^]