1#! /usr/bin/env perl
2# Copyright 2010-2016 The OpenSSL Project Authors. All Rights Reserved.
3#
4# Licensed under the OpenSSL license (the "License").  You may not use
5# this file except in compliance with the License.  You can obtain a copy
6# in the file LICENSE in the source distribution or at
7# https://www.openssl.org/source/license.html
8
9#
10# ====================================================================
11# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL
12# project. The module is, however, dual licensed under OpenSSL and
13# CRYPTOGAMS licenses depending on where you obtain it. For further
14# details see http://www.openssl.org/~appro/cryptogams/.
15# ====================================================================
16#
17# March, May, June 2010
18#
19# The module implements "4-bit" GCM GHASH function and underlying
20# single multiplication operation in GF(2^128). "4-bit" means that it
21# uses 256 bytes per-key table [+64/128 bytes fixed table]. It has two
22# code paths: vanilla x86 and vanilla SSE. Former will be executed on
23# 486 and Pentium, latter on all others. SSE GHASH features so called
24# "528B" variant of "4-bit" method utilizing additional 256+16 bytes
25# of per-key storage [+512 bytes shared table]. Performance results
26# are for streamed GHASH subroutine and are expressed in cycles per
27# processed byte, less is better:
28#
29#		gcc 2.95.3(*)	SSE assembler	x86 assembler
30#
31# Pentium	105/111(**)	-		50
32# PIII		68 /75		12.2		24
33# P4		125/125		17.8		84(***)
34# Opteron	66 /70		10.1		30
35# Core2		54 /67		8.4		18
36# Atom		105/105		16.8		53
37# VIA Nano	69 /71		13.0		27
38#
39# (*)	gcc 3.4.x was observed to generate few percent slower code,
40#	which is one of reasons why 2.95.3 results were chosen,
41#	another reason is lack of 3.4.x results for older CPUs;
42#	comparison with SSE results is not completely fair, because C
43#	results are for vanilla "256B" implementation, while
44#	assembler results are for "528B";-)
45# (**)	second number is result for code compiled with -fPIC flag,
46#	which is actually more relevant, because assembler code is
47#	position-independent;
48# (***)	see comment in non-MMX routine for further details;
49#
50# To summarize, it's >2-5 times faster than gcc-generated code. To
51# anchor it to something else SHA1 assembler processes one byte in
52# ~7 cycles on contemporary x86 cores. As for choice of MMX/SSE
53# in particular, see comment at the end of the file...
54
55# May 2010
56#
57# Add PCLMULQDQ version performing at 2.10 cycles per processed byte.
58# The question is how close is it to theoretical limit? The pclmulqdq
59# instruction latency appears to be 14 cycles and there can't be more
60# than 2 of them executing at any given time. This means that single
61# Karatsuba multiplication would take 28 cycles *plus* few cycles for
62# pre- and post-processing. Then multiplication has to be followed by
63# modulo-reduction. Given that aggregated reduction method [see
64# "Carry-less Multiplication and Its Usage for Computing the GCM Mode"
65# white paper by Intel] allows you to perform reduction only once in
66# a while we can assume that asymptotic performance can be estimated
67# as (28+Tmod/Naggr)/16, where Tmod is time to perform reduction
68# and Naggr is the aggregation factor.
69#
70# Before we proceed to this implementation let's have closer look at
71# the best-performing code suggested by Intel in their white paper.
72# By tracing inter-register dependencies Tmod is estimated as ~19
73# cycles and Naggr chosen by Intel is 4, resulting in 2.05 cycles per
74# processed byte. As implied, this is quite optimistic estimate,
75# because it does not account for Karatsuba pre- and post-processing,
76# which for a single multiplication is ~5 cycles. Unfortunately Intel
77# does not provide performance data for GHASH alone. But benchmarking
78# AES_GCM_encrypt ripped out of Fig. 15 of the white paper with aadt
79# alone resulted in 2.46 cycles per byte of out 16KB buffer. Note that
80# the result accounts even for pre-computing of degrees of the hash
81# key H, but its portion is negligible at 16KB buffer size.
82#
83# Moving on to the implementation in question. Tmod is estimated as
84# ~13 cycles and Naggr is 2, giving asymptotic performance of ...
85# 2.16. How is it possible that measured performance is better than
86# optimistic theoretical estimate? There is one thing Intel failed
87# to recognize. By serializing GHASH with CTR in same subroutine
88# former's performance is really limited to above (Tmul + Tmod/Naggr)
89# equation. But if GHASH procedure is detached, the modulo-reduction
90# can be interleaved with Naggr-1 multiplications at instruction level
91# and under ideal conditions even disappear from the equation. So that
92# optimistic theoretical estimate for this implementation is ...
93# 28/16=1.75, and not 2.16. Well, it's probably way too optimistic,
94# at least for such small Naggr. I'd argue that (28+Tproc/Naggr),
95# where Tproc is time required for Karatsuba pre- and post-processing,
96# is more realistic estimate. In this case it gives ... 1.91 cycles.
97# Or in other words, depending on how well we can interleave reduction
98# and one of the two multiplications the performance should be between
99# 1.91 and 2.16. As already mentioned, this implementation processes
100# one byte out of 8KB buffer in 2.10 cycles, while x86_64 counterpart
101# - in 2.02. x86_64 performance is better, because larger register
102# bank allows to interleave reduction and multiplication better.
103#
104# Does it make sense to increase Naggr? To start with it's virtually
105# impossible in 32-bit mode, because of limited register bank
106# capacity. Otherwise improvement has to be weighed against slower
107# setup, as well as code size and complexity increase. As even
108# optimistic estimate doesn't promise 30% performance improvement,
109# there are currently no plans to increase Naggr.
110#
111# Special thanks to David Woodhouse for providing access to a
112# Westmere-based system on behalf of Intel Open Source Technology Centre.
113
114# January 2010
115#
116# Tweaked to optimize transitions between integer and FP operations
117# on same XMM register, PCLMULQDQ subroutine was measured to process
118# one byte in 2.07 cycles on Sandy Bridge, and in 2.12 - on Westmere.
119# The minor regression on Westmere is outweighed by ~15% improvement
120# on Sandy Bridge. Strangely enough attempt to modify 64-bit code in
121# similar manner resulted in almost 20% degradation on Sandy Bridge,
122# where original 64-bit code processes one byte in 1.95 cycles.
123
124#####################################################################
125# For reference, AMD Bulldozer processes one byte in 1.98 cycles in
126# 32-bit mode and 1.89 in 64-bit.
127
128# February 2013
129#
130# Overhaul: aggregate Karatsuba post-processing, improve ILP in
131# reduction_alg9. Resulting performance is 1.96 cycles per byte on
132# Westmere, 1.95 - on Sandy/Ivy Bridge, 1.76 - on Bulldozer.
133
134$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;
135push(@INC,"${dir}","${dir}../../../perlasm");
136require "x86asm.pl";
137
138$output=pop;
139open STDOUT,">$output";
140
141&asm_init($ARGV[0],$x86only = $ARGV[$#ARGV] eq "386");
142
143$sse2=1;
144
145
146if ($sse2) {{
147######################################################################
148# PCLMULQDQ version.
149
150$Xip="eax";
151$Htbl="edx";
152$const="ecx";
153$inp="esi";
154$len="ebx";
155
156($Xi,$Xhi)=("xmm0","xmm1");	$Hkey="xmm2";
157($T1,$T2,$T3)=("xmm3","xmm4","xmm5");
158($Xn,$Xhn)=("xmm6","xmm7");
159
160&static_label("bswap");
161
162sub clmul64x64_T2 {	# minimal "register" pressure
163my ($Xhi,$Xi,$Hkey,$HK)=@_;
164
165	&movdqa		($Xhi,$Xi);		#
166	&pshufd		($T1,$Xi,0b01001110);
167	&pshufd		($T2,$Hkey,0b01001110)	if (!defined($HK));
168	&pxor		($T1,$Xi);		#
169	&pxor		($T2,$Hkey)		if (!defined($HK));
170			$HK=$T2			if (!defined($HK));
171
172	&pclmulqdq	($Xi,$Hkey,0x00);	#######
173	&pclmulqdq	($Xhi,$Hkey,0x11);	#######
174	&pclmulqdq	($T1,$HK,0x00);		#######
175	&xorps		($T1,$Xi);		#
176	&xorps		($T1,$Xhi);		#
177
178	&movdqa		($T2,$T1);		#
179	&psrldq		($T1,8);
180	&pslldq		($T2,8);		#
181	&pxor		($Xhi,$T1);
182	&pxor		($Xi,$T2);		#
183}
184
185sub clmul64x64_T3 {
186# Even though this subroutine offers visually better ILP, it
187# was empirically found to be a tad slower than above version.
188# At least in GFp_gcm_ghash_clmul context. But it's just as well,
189# because loop modulo-scheduling is possible only thanks to
190# minimized "register" pressure...
191my ($Xhi,$Xi,$Hkey)=@_;
192
193	&movdqa		($T1,$Xi);		#
194	&movdqa		($Xhi,$Xi);
195	&pclmulqdq	($Xi,$Hkey,0x00);	#######
196	&pclmulqdq	($Xhi,$Hkey,0x11);	#######
197	&pshufd		($T2,$T1,0b01001110);	#
198	&pshufd		($T3,$Hkey,0b01001110);
199	&pxor		($T2,$T1);		#
200	&pxor		($T3,$Hkey);
201	&pclmulqdq	($T2,$T3,0x00);		#######
202	&pxor		($T2,$Xi);		#
203	&pxor		($T2,$Xhi);		#
204
205	&movdqa		($T3,$T2);		#
206	&psrldq		($T2,8);
207	&pslldq		($T3,8);		#
208	&pxor		($Xhi,$T2);
209	&pxor		($Xi,$T3);		#
210}
211
212if (1) {		# Algorithm 9 with <<1 twist.
213			# Reduction is shorter and uses only two
214			# temporary registers, which makes it better
215			# candidate for interleaving with 64x64
216			# multiplication. Pre-modulo-scheduled loop
217			# was found to be ~20% faster than Algorithm 5
218			# below. Algorithm 9 was therefore chosen for
219			# further optimization...
220
221sub reduction_alg9 {	# 17/11 times faster than Intel version
222my ($Xhi,$Xi) = @_;
223
224	# 1st phase
225	&movdqa		($T2,$Xi);		#
226	&movdqa		($T1,$Xi);
227	&psllq		($Xi,5);
228	&pxor		($T1,$Xi);		#
229	&psllq		($Xi,1);
230	&pxor		($Xi,$T1);		#
231	&psllq		($Xi,57);		#
232	&movdqa		($T1,$Xi);		#
233	&pslldq		($Xi,8);
234	&psrldq		($T1,8);		#
235	&pxor		($Xi,$T2);
236	&pxor		($Xhi,$T1);		#
237
238	# 2nd phase
239	&movdqa		($T2,$Xi);
240	&psrlq		($Xi,1);
241	&pxor		($Xhi,$T2);		#
242	&pxor		($T2,$Xi);
243	&psrlq		($Xi,5);
244	&pxor		($Xi,$T2);		#
245	&psrlq		($Xi,1);		#
246	&pxor		($Xi,$Xhi)		#
247}
248
249&function_begin_B("GFp_gcm_init_clmul");
250	&mov		($Htbl,&wparam(0));
251	&mov		($Xip,&wparam(1));
252
253	&call		(&label("pic"));
254&set_label("pic");
255	&blindpop	($const);
256	&lea		($const,&DWP(&label("bswap")."-".&label("pic"),$const));
257
258	&movdqu		($Hkey,&QWP(0,$Xip));
259	&pshufd		($Hkey,$Hkey,0b01001110);# dword swap
260
261	# <<1 twist
262	&pshufd		($T2,$Hkey,0b11111111);	# broadcast uppermost dword
263	&movdqa		($T1,$Hkey);
264	&psllq		($Hkey,1);
265	&pxor		($T3,$T3);		#
266	&psrlq		($T1,63);
267	&pcmpgtd	($T3,$T2);		# broadcast carry bit
268	&pslldq		($T1,8);
269	&por		($Hkey,$T1);		# H<<=1
270
271	# magic reduction
272	&pand		($T3,&QWP(16,$const));	# 0x1c2_polynomial
273	&pxor		($Hkey,$T3);		# if(carry) H^=0x1c2_polynomial
274
275	# calculate H^2
276	&movdqa		($Xi,$Hkey);
277	&clmul64x64_T2	($Xhi,$Xi,$Hkey);
278	&reduction_alg9	($Xhi,$Xi);
279
280	&pshufd		($T1,$Hkey,0b01001110);
281	&pshufd		($T2,$Xi,0b01001110);
282	&pxor		($T1,$Hkey);		# Karatsuba pre-processing
283	&movdqu		(&QWP(0,$Htbl),$Hkey);	# save H
284	&pxor		($T2,$Xi);		# Karatsuba pre-processing
285	&movdqu		(&QWP(16,$Htbl),$Xi);	# save H^2
286	&palignr	($T2,$T1,8);		# low part is H.lo^H.hi
287	&movdqu		(&QWP(32,$Htbl),$T2);	# save Karatsuba "salt"
288
289	&ret		();
290&function_end_B("GFp_gcm_init_clmul");
291
292&function_begin_B("GFp_gcm_gmult_clmul");
293	&mov		($Xip,&wparam(0));
294	&mov		($Htbl,&wparam(1));
295
296	&call		(&label("pic"));
297&set_label("pic");
298	&blindpop	($const);
299	&lea		($const,&DWP(&label("bswap")."-".&label("pic"),$const));
300
301	&movdqu		($Xi,&QWP(0,$Xip));
302	&movdqa		($T3,&QWP(0,$const));
303	&movups		($Hkey,&QWP(0,$Htbl));
304	&pshufb		($Xi,$T3);
305	&movups		($T2,&QWP(32,$Htbl));
306
307	&clmul64x64_T2	($Xhi,$Xi,$Hkey,$T2);
308	&reduction_alg9	($Xhi,$Xi);
309
310	&pshufb		($Xi,$T3);
311	&movdqu		(&QWP(0,$Xip),$Xi);
312
313	&ret	();
314&function_end_B("GFp_gcm_gmult_clmul");
315
316&function_begin("GFp_gcm_ghash_clmul");
317	&mov		($Xip,&wparam(0));
318	&mov		($Htbl,&wparam(1));
319	&mov		($inp,&wparam(2));
320	&mov		($len,&wparam(3));
321
322	&call		(&label("pic"));
323&set_label("pic");
324	&blindpop	($const);
325	&lea		($const,&DWP(&label("bswap")."-".&label("pic"),$const));
326
327	&movdqu		($Xi,&QWP(0,$Xip));
328	&movdqa		($T3,&QWP(0,$const));
329	&movdqu		($Hkey,&QWP(0,$Htbl));
330	&pshufb		($Xi,$T3);
331
332	&sub		($len,0x10);
333	&jz		(&label("odd_tail"));
334
335	#######
336	# Xi+2 =[H*(Ii+1 + Xi+1)] mod P =
337	#	[(H*Ii+1) + (H*Xi+1)] mod P =
338	#	[(H*Ii+1) + H^2*(Ii+Xi)] mod P
339	#
340	&movdqu		($T1,&QWP(0,$inp));	# Ii
341	&movdqu		($Xn,&QWP(16,$inp));	# Ii+1
342	&pshufb		($T1,$T3);
343	&pshufb		($Xn,$T3);
344	&movdqu		($T3,&QWP(32,$Htbl));
345	&pxor		($Xi,$T1);		# Ii+Xi
346
347	&pshufd		($T1,$Xn,0b01001110);	# H*Ii+1
348	&movdqa		($Xhn,$Xn);
349	&pxor		($T1,$Xn);		#
350	&lea		($inp,&DWP(32,$inp));	# i+=2
351
352	&pclmulqdq	($Xn,$Hkey,0x00);	#######
353	&pclmulqdq	($Xhn,$Hkey,0x11);	#######
354	&pclmulqdq	($T1,$T3,0x00);		#######
355	&movups		($Hkey,&QWP(16,$Htbl));	# load H^2
356	&nop		();
357
358	&sub		($len,0x20);
359	&jbe		(&label("even_tail"));
360	&jmp		(&label("mod_loop"));
361
362&set_label("mod_loop",32);
363	&pshufd		($T2,$Xi,0b01001110);	# H^2*(Ii+Xi)
364	&movdqa		($Xhi,$Xi);
365	&pxor		($T2,$Xi);		#
366	&nop		();
367
368	&pclmulqdq	($Xi,$Hkey,0x00);	#######
369	&pclmulqdq	($Xhi,$Hkey,0x11);	#######
370	&pclmulqdq	($T2,$T3,0x10);		#######
371	&movups		($Hkey,&QWP(0,$Htbl));	# load H
372
373	&xorps		($Xi,$Xn);		# (H*Ii+1) + H^2*(Ii+Xi)
374	&movdqa		($T3,&QWP(0,$const));
375	&xorps		($Xhi,$Xhn);
376	 &movdqu	($Xhn,&QWP(0,$inp));	# Ii
377	&pxor		($T1,$Xi);		# aggregated Karatsuba post-processing
378	 &movdqu	($Xn,&QWP(16,$inp));	# Ii+1
379	&pxor		($T1,$Xhi);		#
380
381	 &pshufb	($Xhn,$T3);
382	&pxor		($T2,$T1);		#
383
384	&movdqa		($T1,$T2);		#
385	&psrldq		($T2,8);
386	&pslldq		($T1,8);		#
387	&pxor		($Xhi,$T2);
388	&pxor		($Xi,$T1);		#
389	 &pshufb	($Xn,$T3);
390	 &pxor		($Xhi,$Xhn);		# "Ii+Xi", consume early
391
392	&movdqa		($Xhn,$Xn);		#&clmul64x64_TX	($Xhn,$Xn,$Hkey); H*Ii+1
393	  &movdqa	($T2,$Xi);		#&reduction_alg9($Xhi,$Xi); 1st phase
394	  &movdqa	($T1,$Xi);
395	  &psllq	($Xi,5);
396	  &pxor		($T1,$Xi);		#
397	  &psllq	($Xi,1);
398	  &pxor		($Xi,$T1);		#
399	&pclmulqdq	($Xn,$Hkey,0x00);	#######
400	&movups		($T3,&QWP(32,$Htbl));
401	  &psllq	($Xi,57);		#
402	  &movdqa	($T1,$Xi);		#
403	  &pslldq	($Xi,8);
404	  &psrldq	($T1,8);		#
405	  &pxor		($Xi,$T2);
406	  &pxor		($Xhi,$T1);		#
407	&pshufd		($T1,$Xhn,0b01001110);
408	  &movdqa	($T2,$Xi);		# 2nd phase
409	  &psrlq	($Xi,1);
410	&pxor		($T1,$Xhn);
411	  &pxor		($Xhi,$T2);		#
412	&pclmulqdq	($Xhn,$Hkey,0x11);	#######
413	&movups		($Hkey,&QWP(16,$Htbl));	# load H^2
414	  &pxor		($T2,$Xi);
415	  &psrlq	($Xi,5);
416	  &pxor		($Xi,$T2);		#
417	  &psrlq	($Xi,1);		#
418	  &pxor		($Xi,$Xhi)		#
419	&pclmulqdq	($T1,$T3,0x00);		#######
420
421	&lea		($inp,&DWP(32,$inp));
422	&sub		($len,0x20);
423	&ja		(&label("mod_loop"));
424
425&set_label("even_tail");
426	&pshufd		($T2,$Xi,0b01001110);	# H^2*(Ii+Xi)
427	&movdqa		($Xhi,$Xi);
428	&pxor		($T2,$Xi);		#
429
430	&pclmulqdq	($Xi,$Hkey,0x00);	#######
431	&pclmulqdq	($Xhi,$Hkey,0x11);	#######
432	&pclmulqdq	($T2,$T3,0x10);		#######
433	&movdqa		($T3,&QWP(0,$const));
434
435	&xorps		($Xi,$Xn);		# (H*Ii+1) + H^2*(Ii+Xi)
436	&xorps		($Xhi,$Xhn);
437	&pxor		($T1,$Xi);		# aggregated Karatsuba post-processing
438	&pxor		($T1,$Xhi);		#
439
440	&pxor		($T2,$T1);		#
441
442	&movdqa		($T1,$T2);		#
443	&psrldq		($T2,8);
444	&pslldq		($T1,8);		#
445	&pxor		($Xhi,$T2);
446	&pxor		($Xi,$T1);		#
447
448	&reduction_alg9	($Xhi,$Xi);
449
450	&test		($len,$len);
451	&jnz		(&label("done"));
452
453	&movups		($Hkey,&QWP(0,$Htbl));	# load H
454&set_label("odd_tail");
455	&movdqu		($T1,&QWP(0,$inp));	# Ii
456	&pshufb		($T1,$T3);
457	&pxor		($Xi,$T1);		# Ii+Xi
458
459	&clmul64x64_T2	($Xhi,$Xi,$Hkey);	# H*(Ii+Xi)
460	&reduction_alg9	($Xhi,$Xi);
461
462&set_label("done");
463	&pshufb		($Xi,$T3);
464	&movdqu		(&QWP(0,$Xip),$Xi);
465&function_end("GFp_gcm_ghash_clmul");
466
467} else {		# Algorithm 5. Kept for reference purposes.
468
469sub reduction_alg5 {	# 19/16 times faster than Intel version
470my ($Xhi,$Xi)=@_;
471
472	# <<1
473	&movdqa		($T1,$Xi);		#
474	&movdqa		($T2,$Xhi);
475	&pslld		($Xi,1);
476	&pslld		($Xhi,1);		#
477	&psrld		($T1,31);
478	&psrld		($T2,31);		#
479	&movdqa		($T3,$T1);
480	&pslldq		($T1,4);
481	&psrldq		($T3,12);		#
482	&pslldq		($T2,4);
483	&por		($Xhi,$T3);		#
484	&por		($Xi,$T1);
485	&por		($Xhi,$T2);		#
486
487	# 1st phase
488	&movdqa		($T1,$Xi);
489	&movdqa		($T2,$Xi);
490	&movdqa		($T3,$Xi);		#
491	&pslld		($T1,31);
492	&pslld		($T2,30);
493	&pslld		($Xi,25);		#
494	&pxor		($T1,$T2);
495	&pxor		($T1,$Xi);		#
496	&movdqa		($T2,$T1);		#
497	&pslldq		($T1,12);
498	&psrldq		($T2,4);		#
499	&pxor		($T3,$T1);
500
501	# 2nd phase
502	&pxor		($Xhi,$T3);		#
503	&movdqa		($Xi,$T3);
504	&movdqa		($T1,$T3);
505	&psrld		($Xi,1);		#
506	&psrld		($T1,2);
507	&psrld		($T3,7);		#
508	&pxor		($Xi,$T1);
509	&pxor		($Xhi,$T2);
510	&pxor		($Xi,$T3);		#
511	&pxor		($Xi,$Xhi);		#
512}
513
514&function_begin_B("GFp_gcm_init_clmul");
515	&mov		($Htbl,&wparam(0));
516	&mov		($Xip,&wparam(1));
517
518	&call		(&label("pic"));
519&set_label("pic");
520	&blindpop	($const);
521	&lea		($const,&DWP(&label("bswap")."-".&label("pic"),$const));
522
523	&movdqu		($Hkey,&QWP(0,$Xip));
524	&pshufd		($Hkey,$Hkey,0b01001110);# dword swap
525
526	# calculate H^2
527	&movdqa		($Xi,$Hkey);
528	&clmul64x64_T3	($Xhi,$Xi,$Hkey);
529	&reduction_alg5	($Xhi,$Xi);
530
531	&movdqu		(&QWP(0,$Htbl),$Hkey);	# save H
532	&movdqu		(&QWP(16,$Htbl),$Xi);	# save H^2
533
534	&ret		();
535&function_end_B("GFp_gcm_init_clmul");
536
537&function_begin_B("GFp_gcm_gmult_clmul");
538	&mov		($Xip,&wparam(0));
539	&mov		($Htbl,&wparam(1));
540
541	&call		(&label("pic"));
542&set_label("pic");
543	&blindpop	($const);
544	&lea		($const,&DWP(&label("bswap")."-".&label("pic"),$const));
545
546	&movdqu		($Xi,&QWP(0,$Xip));
547	&movdqa		($Xn,&QWP(0,$const));
548	&movdqu		($Hkey,&QWP(0,$Htbl));
549	&pshufb		($Xi,$Xn);
550
551	&clmul64x64_T3	($Xhi,$Xi,$Hkey);
552	&reduction_alg5	($Xhi,$Xi);
553
554	&pshufb		($Xi,$Xn);
555	&movdqu		(&QWP(0,$Xip),$Xi);
556
557	&ret	();
558&function_end_B("GFp_gcm_gmult_clmul");
559
560&function_begin("GFp_gcm_ghash_clmul");
561	&mov		($Xip,&wparam(0));
562	&mov		($Htbl,&wparam(1));
563	&mov		($inp,&wparam(2));
564	&mov		($len,&wparam(3));
565
566	&call		(&label("pic"));
567&set_label("pic");
568	&blindpop	($const);
569	&lea		($const,&DWP(&label("bswap")."-".&label("pic"),$const));
570
571	&movdqu		($Xi,&QWP(0,$Xip));
572	&movdqa		($T3,&QWP(0,$const));
573	&movdqu		($Hkey,&QWP(0,$Htbl));
574	&pshufb		($Xi,$T3);
575
576	&sub		($len,0x10);
577	&jz		(&label("odd_tail"));
578
579	#######
580	# Xi+2 =[H*(Ii+1 + Xi+1)] mod P =
581	#	[(H*Ii+1) + (H*Xi+1)] mod P =
582	#	[(H*Ii+1) + H^2*(Ii+Xi)] mod P
583	#
584	&movdqu		($T1,&QWP(0,$inp));	# Ii
585	&movdqu		($Xn,&QWP(16,$inp));	# Ii+1
586	&pshufb		($T1,$T3);
587	&pshufb		($Xn,$T3);
588	&pxor		($Xi,$T1);		# Ii+Xi
589
590	&clmul64x64_T3	($Xhn,$Xn,$Hkey);	# H*Ii+1
591	&movdqu		($Hkey,&QWP(16,$Htbl));	# load H^2
592
593	&sub		($len,0x20);
594	&lea		($inp,&DWP(32,$inp));	# i+=2
595	&jbe		(&label("even_tail"));
596
597&set_label("mod_loop");
598	&clmul64x64_T3	($Xhi,$Xi,$Hkey);	# H^2*(Ii+Xi)
599	&movdqu		($Hkey,&QWP(0,$Htbl));	# load H
600
601	&pxor		($Xi,$Xn);		# (H*Ii+1) + H^2*(Ii+Xi)
602	&pxor		($Xhi,$Xhn);
603
604	&reduction_alg5	($Xhi,$Xi);
605
606	#######
607	&movdqa		($T3,&QWP(0,$const));
608	&movdqu		($T1,&QWP(0,$inp));	# Ii
609	&movdqu		($Xn,&QWP(16,$inp));	# Ii+1
610	&pshufb		($T1,$T3);
611	&pshufb		($Xn,$T3);
612	&pxor		($Xi,$T1);		# Ii+Xi
613
614	&clmul64x64_T3	($Xhn,$Xn,$Hkey);	# H*Ii+1
615	&movdqu		($Hkey,&QWP(16,$Htbl));	# load H^2
616
617	&sub		($len,0x20);
618	&lea		($inp,&DWP(32,$inp));
619	&ja		(&label("mod_loop"));
620
621&set_label("even_tail");
622	&clmul64x64_T3	($Xhi,$Xi,$Hkey);	# H^2*(Ii+Xi)
623
624	&pxor		($Xi,$Xn);		# (H*Ii+1) + H^2*(Ii+Xi)
625	&pxor		($Xhi,$Xhn);
626
627	&reduction_alg5	($Xhi,$Xi);
628
629	&movdqa		($T3,&QWP(0,$const));
630	&test		($len,$len);
631	&jnz		(&label("done"));
632
633	&movdqu		($Hkey,&QWP(0,$Htbl));	# load H
634&set_label("odd_tail");
635	&movdqu		($T1,&QWP(0,$inp));	# Ii
636	&pshufb		($T1,$T3);
637	&pxor		($Xi,$T1);		# Ii+Xi
638
639	&clmul64x64_T3	($Xhi,$Xi,$Hkey);	# H*(Ii+Xi)
640	&reduction_alg5	($Xhi,$Xi);
641
642	&movdqa		($T3,&QWP(0,$const));
643&set_label("done");
644	&pshufb		($Xi,$T3);
645	&movdqu		(&QWP(0,$Xip),$Xi);
646&function_end("GFp_gcm_ghash_clmul");
647
648}
649
650&set_label("bswap",64);
651	&data_byte(15,14,13,12,11,10,9,8,7,6,5,4,3,2,1,0);
652	&data_byte(1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2);	# 0x1c2_polynomial
653&set_label("rem_8bit",64);
654	&data_short(0x0000,0x01C2,0x0384,0x0246,0x0708,0x06CA,0x048C,0x054E);
655	&data_short(0x0E10,0x0FD2,0x0D94,0x0C56,0x0918,0x08DA,0x0A9C,0x0B5E);
656	&data_short(0x1C20,0x1DE2,0x1FA4,0x1E66,0x1B28,0x1AEA,0x18AC,0x196E);
657	&data_short(0x1230,0x13F2,0x11B4,0x1076,0x1538,0x14FA,0x16BC,0x177E);
658	&data_short(0x3840,0x3982,0x3BC4,0x3A06,0x3F48,0x3E8A,0x3CCC,0x3D0E);
659	&data_short(0x3650,0x3792,0x35D4,0x3416,0x3158,0x309A,0x32DC,0x331E);
660	&data_short(0x2460,0x25A2,0x27E4,0x2626,0x2368,0x22AA,0x20EC,0x212E);
661	&data_short(0x2A70,0x2BB2,0x29F4,0x2836,0x2D78,0x2CBA,0x2EFC,0x2F3E);
662	&data_short(0x7080,0x7142,0x7304,0x72C6,0x7788,0x764A,0x740C,0x75CE);
663	&data_short(0x7E90,0x7F52,0x7D14,0x7CD6,0x7998,0x785A,0x7A1C,0x7BDE);
664	&data_short(0x6CA0,0x6D62,0x6F24,0x6EE6,0x6BA8,0x6A6A,0x682C,0x69EE);
665	&data_short(0x62B0,0x6372,0x6134,0x60F6,0x65B8,0x647A,0x663C,0x67FE);
666	&data_short(0x48C0,0x4902,0x4B44,0x4A86,0x4FC8,0x4E0A,0x4C4C,0x4D8E);
667	&data_short(0x46D0,0x4712,0x4554,0x4496,0x41D8,0x401A,0x425C,0x439E);
668	&data_short(0x54E0,0x5522,0x5764,0x56A6,0x53E8,0x522A,0x506C,0x51AE);
669	&data_short(0x5AF0,0x5B32,0x5974,0x58B6,0x5DF8,0x5C3A,0x5E7C,0x5FBE);
670	&data_short(0xE100,0xE0C2,0xE284,0xE346,0xE608,0xE7CA,0xE58C,0xE44E);
671	&data_short(0xEF10,0xEED2,0xEC94,0xED56,0xE818,0xE9DA,0xEB9C,0xEA5E);
672	&data_short(0xFD20,0xFCE2,0xFEA4,0xFF66,0xFA28,0xFBEA,0xF9AC,0xF86E);
673	&data_short(0xF330,0xF2F2,0xF0B4,0xF176,0xF438,0xF5FA,0xF7BC,0xF67E);
674	&data_short(0xD940,0xD882,0xDAC4,0xDB06,0xDE48,0xDF8A,0xDDCC,0xDC0E);
675	&data_short(0xD750,0xD692,0xD4D4,0xD516,0xD058,0xD19A,0xD3DC,0xD21E);
676	&data_short(0xC560,0xC4A2,0xC6E4,0xC726,0xC268,0xC3AA,0xC1EC,0xC02E);
677	&data_short(0xCB70,0xCAB2,0xC8F4,0xC936,0xCC78,0xCDBA,0xCFFC,0xCE3E);
678	&data_short(0x9180,0x9042,0x9204,0x93C6,0x9688,0x974A,0x950C,0x94CE);
679	&data_short(0x9F90,0x9E52,0x9C14,0x9DD6,0x9898,0x995A,0x9B1C,0x9ADE);
680	&data_short(0x8DA0,0x8C62,0x8E24,0x8FE6,0x8AA8,0x8B6A,0x892C,0x88EE);
681	&data_short(0x83B0,0x8272,0x8034,0x81F6,0x84B8,0x857A,0x873C,0x86FE);
682	&data_short(0xA9C0,0xA802,0xAA44,0xAB86,0xAEC8,0xAF0A,0xAD4C,0xAC8E);
683	&data_short(0xA7D0,0xA612,0xA454,0xA596,0xA0D8,0xA11A,0xA35C,0xA29E);
684	&data_short(0xB5E0,0xB422,0xB664,0xB7A6,0xB2E8,0xB32A,0xB16C,0xB0AE);
685	&data_short(0xBBF0,0xBA32,0xB874,0xB9B6,0xBCF8,0xBD3A,0xBF7C,0xBEBE);
686}}	# $sse2
687
688&asciz("GHASH for x86, CRYPTOGAMS by <appro\@openssl.org>");
689&asm_finish();
690
691close STDOUT or die "error closing STDOUT";
692
693# A question was risen about choice of vanilla MMX. Or rather why wasn't
694# SSE2 chosen instead? In addition to the fact that MMX runs on legacy
695# CPUs such as PIII, "4-bit" MMX version was observed to provide better
696# performance than *corresponding* SSE2 one even on contemporary CPUs.
697# SSE2 results were provided by Peter-Michael Hager. He maintains SSE2
698# implementation featuring full range of lookup-table sizes, but with
699# per-invocation lookup table setup. Latter means that table size is
700# chosen depending on how much data is to be hashed in every given call,
701# more data - larger table. Best reported result for Core2 is ~4 cycles
702# per processed byte out of 64KB block. This number accounts even for
703# 64KB table setup overhead. As discussed in gcm128.c we choose to be
704# more conservative in respect to lookup table sizes, but how do the
705# results compare? Minimalistic "256B" MMX version delivers ~11 cycles
706# on same platform. As also discussed in gcm128.c, next in line "8-bit
707# Shoup's" or "4KB" method should deliver twice the performance of
708# "256B" one, in other words not worse than ~6 cycles per byte. It
709# should be also be noted that in SSE2 case improvement can be "super-
710# linear," i.e. more than twice, mostly because >>8 maps to single
711# instruction on SSE2 register. This is unlike "4-bit" case when >>4
712# maps to same amount of instructions in both MMX and SSE2 cases.
713# Bottom line is that switch to SSE2 is considered to be justifiable
714# only in case we choose to implement "8-bit" method...
715