| blob | 6b8ee413ff0d20277f0101fbc5ab0174a6cc1b30 |
2 <article>
4 <!-- Title information -->
8 <abstract>
9 This document investigates the possibility of adding scaled window
10 support to the DMX X server, thereby allowing a window or some
11 selected part of the logical DMX area to be displayed using a
12 scaling factor. For example, this might allow the contents of a
13 window to be magnified for easier viewing. In particular, scaling
15 by Red Hat, Inc., Raleigh, North Carolina</it>
16 </abstract>
18 <!-- Table of contents -->
19 <toc>
21 <!-- Begin the document -->
22 <sect>Introduction
23 <sect1>DMX
24 <p>
25 The DMX X server (Xdmx) is a proxy server that is designed
26 to allow X servers on multiple machines to be combined into
27 a single multi-headed X server. Combined with Xinerama,
28 these heads can appear as a single very high-resolution
29 screen. Typical applications include the creation of a
30 video wall with 16 1280x1024 displays arranged in a
31 rectangle, for a total resolution of of 5120x4096.
32 </p>
33 </sect1>
34 <sect1>Problem Statement
35 <p>
36 Applications displayed on a physically large video wall that
37 provides high pixel-resolution may be difficult to see,
38 especially if the application is designed for use on a
39 typical desktop computer with a relatively small display
40 located close to the human operator. The goal of this paper
41 is to describe and discuss solutions to this problem.
42 </p>
43 <p>
44 The original driving problem for this work is to provide
46 displayed using DMX (VNC scaling is currently available only
47 with the Windows client, and there is no plan to extend that
48 capability to other clients). While this specific problem
49 will be addressed in this paper, the general solution space
50 will also be explored, since this may lead to a good
52 other applications.
53 </p>
54 </sect1>
55 <sect1>Task
56 <p>
57 For reference, here is the original description of the task
58 this paper addresses:
59 <itemize>
60 <item>Scaled window support (for VNC)
61 <itemize>
62 <item>
63 Investigate possibility of implementing a "scaled
64 window" extension:
65 <itemize>
66 <item>
67 Add XCreateScaledWindow call that could be used
68 in place of XCreateWindow
69 </item>
70 <item>
71 All primitives drawn to scaled window would be
72 scaled by appropriate (integral?) scaling factor
73 </item>
74 </itemize>
75 </item>
76 <item>
77 Alternate approach: special case VNC support
78 </item>
79 </itemize>
80 </item>
81 </itemize>
82 </p>
83 </sect1>
84 </sect>
86 <sect>Previous Work
87 <p>
88 This section reviews relevant previous work.
89 </p>
90 <sect1>VNC
91 <sect2>Scaling under VNC
92 <p>
94 is possible to specify a scaling factor (as numerator and
95 denominator). When scaling is in effect, the viewer
96 software uses StretchBlt (instead of BitBlt) to display
97 the pixels for the user. When this call is made, the
98 viewer already has received all of the pixel information
99 (at full unscaled resolution).
100 </p>
101 <p>
102 The scaling in VNC is primitive. It does not conserve
103 bandwidth, it does not treat textual information
104 differently (i.e., by using a suitably scaled font), and
105 it does not provide any anti-aliasing other than that
106 provided by the underlying (Windows-only) system library.
107 </p>
108 </sect2>
109 </sect1>
110 <sect1>The X Video Extension
111 <p>
112 The X Video Extension is a widely-available extension to the
113 X11 protocol that provides support for streaming video.
114 Integral to this support is the ability to arbitrarily scale
115 the output. In version 2.2 of the X Video specification,
116 support for scaled still images was provided, using both
117 shared memory and traditional transport. The API for this
118 support uses calls that are quite similar to XCreateWindow,
119 XPutImage, and XShmPutImage. Currently, most of the drivers
120 implemented in XFree86 only support data in various YUV
121 formats. However, several modern video adaptors support RGB
122 as well.
123 </p>
124 <p>
125 Note, though, that the target output for this scaling is an
126 overlay plane -- so X Video provides functionality that is
127 fundamentally different from that provided by the Windows
128 StrechBlt call.
129 </p>
130 </sect1>
131 </sect>
133 <sect>Possible Solutions
134 <p>
135 This section briefly discusses possible solutions, including
136 major advantages and disadvantages from both the
137 implementation and the end-user programmer standpoint.
138 </p>
139 <sect1>VNC-like Scaling
140 <sect2>Software Scaling
141 <p>
143 provide software scaling. This is not a general solution,
144 but it does solve one of the goals of this work.
145 </p>
146 <p>
147 A prototype of this solution was implemented and a patch
150 available for this work, all of the edge cases were not
151 considered and the solution works well mainly for integer
152 scaling.
153 </p>
154 <p>
156 with XPutImage, XCopyArea, and XFillRectangle. All
157 instances of these calls have to be aware of scaling
158 and must round correctly. In the prototype solution,
159 rounding is incorrect and can cause artifacts.
160 </p>
161 <p>
162 A better solution would be to cache all updates to the
164 damaged area to the X display with XPutImage. This would
165 allow the damaged area to be computed so that rounding
166 errors do not create artifacts. This method is probably
167 similar to what is used in the Window client. (The whole
168 VNC suite is being re-written in C++ and the forthcoming
169 version 4 has not been evaluated.)
170 </p>
171 </sect2>
172 <sect2>Scaling with the X Video Extension
173 <p>
175 makes use of a scaled blit that is supplied by the
176 underlying system library. Several video cards currently
177 provide support for a scaled blit, and some X servers
178 (including XFree86) expose this capability to applications
179 via the XvPutImage interface of the X Video Extension.
180 The capability exposed by XvPutImage results in the scaled
181 image being drawn to an overlay plane. Most video cards
182 also provide support for a scaled blit into the normal
183 output planes, but this is not exposed via XvPutImage.
184 </p>
185 <p>
187 the X Video Extension to provide scaling under X11 that is
188 similar to the scaling currently provided under Windows.
189 Unfortunately, Xdmx does not currently export the X Video
190 Extension, so this would not provide an immediate solution
191 usable with DMX.
192 </p>
193 <p>
194 A very early-stage proof-of-concept prototype was
195 implemented and a preliminary patch against
198 implemented to better understand the problems that must be
199 solved to make this solution viable:
200 <itemize>
201 <item>
202 As noted under the software scaling section above,
204 several different calls. These calls write to the
205 normal output planes and are compatible with
206 XvPutImage, which writes to an overlay plane. To
207 eliminate artifacts caused by this problem,
209 copy of the desktop is available, either as a
210 client-side image or a server-side off-screen pixmap,
211 so that XvPutImage would be the only method for
212 writing to the X display.
213 </item>
214 <item>
215 <p>
216 Although several modern graphics adaptors support
217 hardware scaling using an RGB format (e.g., ATI
218 Radeon, nVidia, etc.), XFree86 drivers typically
219 only implement YUV formats. YUV generally compress
220 the pixel information in some way. For example, two
221 commonly implemented formats, YUY2 and UYVY provide
222 intensity information for every RGB pixel, but only
223 provide chroma and luminance information for pairs
224 of horizontal pixels. Since VNC uses
225 pixel-resolution for communicating updates on the
226 wire, additional artifacts are introduced (because
227 there may not be enough information from the wire to
228 update a pair of pixels).
229 <p>
230 Further, the well-known problem with YUV encoding
231 is even more evident when the image is a desktop
232 instead of a movie. For example, consider a
233 1-pixel-wide vertical window border. If the border
234 changes in color but not intensity (e.g., because a
235 window manager uses color to indicate focus), there
236 may or may not be a change in the YUY2 image,
237 depending on the algorithm used for RGB to YUV
238 conversion and on how the border pixel is ordered in
239 the pair of pixels used by the algorithm.
240 <p>
241 Many of these artifacts could be eliminated if
243 the desktop, and only did the conversion to YUV for
244 properly aligned areas of damage. The remaining artifacts
245 could be eliminated if an RGB format was used with X
246 Video (which may require the extension of existing
247 XFree86 drivers to support RGB).
248 </item>
249 <item>
250 Most modern video cards support exactly one overlay
251 plane that is suitable for use with X Video.
252 Therefore, only one application can use X Video at any
253 given time. This is a severe limitation in a desktop
254 environment.
255 </item>
256 </itemize>
257 </p>
258 <sect3>Implementing the X Video Extension for DMX
259 <p>
260 The user-level API for X Video is fairly simple, but the
261 underlying support required for the full specification
262 is large. However, since the API provides a method to
263 query supported capabilities, a usable subset of X
264 Video can be implemented that would support XvPutImage
265 and little else. This would require support for the
266 following:
267 <itemize>
268 <item>
269 X Video Extension API calls, including the
270 following:
271 <itemize>
282 </itemize>
283 </item>
284 <item>
285 Support for querying back-end X Video Extension
286 capabilities.
287 </item>
288 <item>
289 Support for sending the image to the back-ends.
290 Because X Video requires sending full images, there
291 may be a trade-off between bandwidth limitations and
292 additional complexity to divide the image up such
293 that is scales properly.
294 </item>
295 <item>
296 Possible support for a software fall-back. For
297 example, if all of the back-ends do not support the X
298 Video Extension, software scaling can be implemented
299 such that the image is sent to the back-end with
300 XPutImage. This pathway would have poor
301 performance.
302 </item>
303 </itemize>
304 </p>
305 </sect3>
306 <sect3>Supporting RGB formats for the X Video Extension
307 <p>
308 Assuming an XFree86 driver already supports the X Video
309 Extension, and assuming the target hardware supports an
310 RGB format, then adding support for that format is
311 relatively simple and straightforward.
312 </p>
313 </sect3>
314 </sect2>
315 <sect2>Scaling with an XPutImageScaled Extension
316 <p>
317 Instead of (or in addition to) implementing the X Video
318 Extension in DMX, one obvious solution would be to
319 implement a new extension that provides access to
320 hardware-assisted scaled blits, similar to the StretchBlt
321 call available under Windows. This call would scale RGB
322 images and would not use the overlay plane (unlike the X
323 Video Extension).
324 </p>
325 <p>
326 This approach has many of the same advantages and
327 disadvantages as the XCopyAreaScaled Extension, discussed
328 in the next section. Discussion of XPutImageScaled is
329 deferred in favor of XCopyAreaScaled for the following
330 reasons:
331 <itemize>
332 <item>
333 XPutImageScaled can be emulated with XCopyAreaScaled
334 by first using XPutImage to copy the image to an
335 off-screen pixmap, and then calling XCopyAreaScaled
336 between that off-screen pixmap and the target
337 drawable.
338 </item>
339 <item>
340 Since XCopyAreaScaled would copy between two areas of
341 on-screen or off-screen memory, it has additional uses
342 and can be viewed as efficiently providing a superset
343 of XPutImageScaled functionality.
344 </item>
345 </itemize>
346 </p>
347 </sect2>
348 <sect2>Scaling with an XCopyAreaScaled Extension
349 <p>
350 As noted in the previous section, because XCopyAreaScaled
351 provides a superset of the functionality provided by
352 XPutImageScaled, we will consider this extension instead.
353 </p>
354 <p>
355 First, XCopyAreaScaled would provide for RGB scaling
356 between pixmaps (i.e., on-screen or off-screen areas of
357 memory that reside on the video card). Unlike the X Video
358 Extension, which writes into an overlay plane,
359 XCopyAreaScaled would write into the non-overlay areas of
360 the screen. Key points to consider are as follows:
361 <itemize>
362 <item>
363 Because different planes are involved, the two scaling
364 operations are usually implemented in hardware
365 differently, so an XCopyAreaScaled extension could be
366 added in a manner that would neither conflict with nor
367 interact with the X Video extension in any way.
368 </item>
369 <item>
370 The XCopyAreaScaled extension provides new
371 functionality that the X Video Extension does not
372 provide. Based on anecdotal feedback, we believe that
373 many people outside the DMX and VNC communities would
374 be excited about this extension.
375 </item>
376 <item>
377 The main drawback to this extension is that it is new
378 and needs to be implemented at the driver level in
379 XFree86 for each video card to be supported. At the
380 present time, it is more likely that the X Video
381 Extension will be implemented for a particular piece
382 hardware because the X Video extension has multimedia
383 uses. However, over time, we would expect the
384 XCopyAreaScaled extension to be implemented along with
385 the X Video extension, especially if it becomes
386 popular.
387 </item>
388 <item>
389 Another drawback is that not all modern cards provide
390 support for a simple scaled blit operation. However,
391 these cards usually do provide a 3D pipeline which
392 could be used to provide this functionality in a
393 manner that is transparent to the client application
394 that is using the XCopyAreaScaled extension. However,
395 this implementation pathway would make this extension
396 somewhat more difficult to implement on certain cards.
397 </item>
398 </itemize>
399 </p>
400 </sect2>
401 <sect2>Scaling with OpenGL
402 <p>
403 Another general solution to the scaling problem is to use
404 the texture scaling found in all 3D hardware. This
405 ability is already exposed through OpenGL and can be
406 exploited by clients without X server modification (i.e.,
407 other than the ability to support OpenGL). An application
408 using OpenGL would transmit the non-scaled image to the X
409 server as a texture, and would then display a single
410 non-transformed rect using that texture. This also works
411 around the single overlay problem with the X Video
412 Extension as well as the need to implement additional
413 scaled primitive extensions.
414 </p>
415 <p>
416 The downside is that most OpenGL implementations require
417 power of 2 texture sizes and this can be very wasteful of
418 memory if, for example, the application needs to scale a
421 texture). Another downside is that some OpenGL
422 implementations have a limited about of texture memory and
423 cannot handle textures that are very large. For example,
424 they might limit the texture size to 1024x1024.
425 </p>
426 </sect2>
427 </sect1>
428 <sect1>Application-transparent Scaling for DMX
429 <sect2>Back-end Scaling Without Disconnect/Reconnect
430 <p>
431 VNC does scaling on the client side (in the
433 solution for DMX would require support in the back-end X
434 servers and, therefore, is not a general solution.
435 </p>
436 <p>
437 XFree86 already implements some support for "scaling" that
438 could be used with DMX: if, in the XF86Config file,
439 multiple Modes are listed in the Display Subsection of the
440 Screen Section, then pressing Ctrl-Alt-Plus and
441 Ctrl-Alt-Minus can be used to iterate through the listed
442 modes. The display dimensions will change to the
443 dimensions in the Modes line, but the logical dimensions
444 of the X server (i.e., the dimensions that Xdmx knows
445 about) will not change.
446 </p>
447 <p>
448 Further, the dimensions of the XFree86 display are under
449 software control (via the XFree86-VidModeExtension), so
450 the Xdmx server could change the screen dimensions on a
451 per-display basis, thereby scaling the information on part
452 of that display.
453 </p>
454 <p>
455 However, this scaling appears to have limited use. For
457 1280x1024 displays. If all of the back-end servers were
458 simultaneously configured to display 640x480, the left
459 hand corner of each display would be magnified, but the
460 composite result would be unreadable. Magnifying one
461 display at a time could be usable, but could have limited
462 utility, since the result would still be no larger than a
463 single display.
464 </p>
465 </sect2>
466 <sect2>Back-end Scaling With Disconnect/Reconnect
467 <p>
468 Disconnect and reconnect features are not currently
469 supported in DMX, but are scheduled to be implemented in
470 the future. These features, combined with the
471 XFree86-VidModeExtension Extension, would allow an
472 application to do the following:
473 <itemize>
474 <item>
475 Disconnect a specific back-end server (via the DMX
476 Extension),
477 </item>
478 <item>
479 reconfigure the XFree86 back-end server resolution,
480 and
481 </item>
482 <item>
483 reconnect the back-end server to DMX -- at a new
484 origin with the new screen resolution.
485 </item>
486 </itemize>
487 </p>
488 <p>
489 For example, consider a display wall consisting of 16
491 All of the screens could be disconnected, repositioned,
492 and reconnected each at a resolution of 640x480. The
493 total resolution of the display wall would be 2560x1920,
494 allowing a view of a selected area approximately
495 one-fourth of the size of the DMX display. This change
496 would be completely application independent (except,
497 perhaps, for a DMX-aware window manager). When work at
498 the increased resolution was completed, the back-end
499 servers could be disconnected, reconfigured, and
500 reconnected for the original 5120x4096 view.
501 </p>
502 <p>
503 Support for this type of scaling can be implemented in a
504 DMX-aware X11 client assuming the DMX server support
505 arbitrary disconnect and reconnect semantics. Because
506 this application cannot be written before
507 disconnect/reconnect is implemented, this solution will
508 not be discussed further in this paper.
509 </p>
510 </sect2>
511 <sect2>Server-side Scaling
512 <p>
513 In earlier versions of DMX, a frame buffer was maintained
514 on the server side, and XPutImage was used to move the
515 information from the server to the client (similar to some
516 early VNC implementations). The use of a server-side
517 frame buffer would allow the server to do scaling, but is
518 not a recommended solution because of overall performance
519 issues and server-side memory issues (i.e., the frame
520 buffer would be very large for large display walls).
521 </p>
522 <p>
523 Exploration of this path is not recommended.
524 </p>
525 </sect2>
526 </sect1>
527 <sect1>XCreateScaledWindow API
528 <p>
529 The implementation of X Video Extension in DMX, and the use
530 of XvPutImage by applications requiring scaling requires
531 significant changes in DMX Further, XvPutImage is,
532 essentially a scaled blit, and it is only useful for
533 applications which are already using (or can be modified to
534 use) XPutImage. Therefore, a more general API will be
535 discussed as another possibility.
536 </p>
537 <p>
538 X applications typically create windows with the
539 XCreateWindow call. A new extension could provide an
540 XCreateScaledWindow call that could be used in place of the
541 XCreateWindow call and be otherwise transparent to the
542 application. This would allow applications, even those that
543 do not depend on XPutImage, to take advantage of window
544 scaling. In this section we describe how the call would
545 work, what transparency it provides, and how to solve the
546 potential problems that transparency creates.
547 </p>
548 <sect2>XCreateWindow
549 <p>
550 The XCreateWindow call takes width and height as
551 parameters. An XCreateScaledWindow call could take all
552 the same parameters, with the addition of a scaling factor.
553 </p>
554 </sect2>
555 <sect2>XSetWindowAttributes
556 <p>
557 An X11 window has several attributes that would have to be
558 scaled:
559 <itemize>
563 </itemize>
564 </p>
565 </sect2>
566 <sect2>XGetWindowAttributes, XGetGeometry
567 <p>
568 For transparency, calls that query the window attributes
569 should return unscaled information. This suggests that
570 all unscaled pixmaps and window attributes should be
571 cached.
572 </p>
573 <p>
574 Unfortunately, a window manager requires the scaled
575 geometry to properly decorate the window. The X server
576 can probably determine which client is acting as the
577 window manager (e.g., because that client will select
578 events that are used exclusively by the window manager).
579 However, other Scaled Window Extension aware clients may
580 also need to determine the scaled geometry. Therefore, at
581 least two additional extension calls should be
582 implemented: XGetScaledWindowAttributes and
583 XGetScaledGeometry.
584 </p>
585 </sect2>
586 <sect2>Popup and Child window positions
587 <p>
588 Some applications may position popup and child windows
589 based on an unscaled notion of the main window geometry.
590 In this case, additional modifications to the client would
591 be required.
592 </p>
593 </sect2>
594 <sect2>Events
595 <p>
596 Most events (e.g., for mouse motion) return information
597 about the coordinates at which the even occurred. These
598 coordinates would have to be modified so that unscaled
599 values were presented to the client.
600 </p>
601 </sect2>
602 <sect2>Implementation
603 <p>
604 There are many implementation issues, some of which are
605 similar to the issues involved in implementing the X Video
606 Extension for DMX. The window contents must be scaled,
607 either by performing all operations to a frame buffer and
608 then writing the image to the display (perhaps using
609 hardware scaling support), or by modifying all of the
610 various drawing operations to perform scaling. Because of
611 the complexity involved, the frame buffer option is
612 recommended.
613 </p>
614 </sect2>
615 </sect1>
616 </sect>
618 <sect>Conclusion and Recommendations
619 <p>
620 We recommend a three phase implementation strategy, based on
621 how an application could be written to take advantage of
622 scaling:
623 <enum>
624 <item>
625 <p>
626 The XCopyAreaScaled extension should be implemented, since
627 this is the ideal solution for applications like VNC, and
628 since making use of this extension will require minimal
629 changes to applications that already use XPutImage or
630 XCopyArea.
631 <p>
632 The initial implementation work would include the design
633 of the X protocol extension, writing this up in the
634 usual format for extension documentation, implementation
635 of the protocol transport pieces in XFree86,
636 implementation of a software fall-back in XFree86 and
637 DMX, one example hardware implementation for XFree86,
638 and implementation of support for this extension in DMX.
639 <p>
640 We suggest implementing the extension first on the ATI
641 Radeon cards. However, since these cards do not provide
642 a 2D scaled blit primitive, the implementation would
643 have to make use of the 3D texture engine to emulate a
644 scaled blit. This is recommended, since other modern
645 graphics cards also do not provide a simple 2D scaled
646 blit operation and an example of the more difficult
647 implementation pathway would be helpful to others.
648 </item>
649 <item>
650 <p>
651 Until XCopyAreaScaled is widely supported, applications
652 that require scaling will have to fall back to another
653 scaling method. We suggest OpenGL as the first fall-back
654 method because it is widely available and supported by
655 DMX.
656 <p>
657 A project centered around OpenGL-based scaling would
658 implement this scaling in VNC as an example. This work
660 rendering engine to cache a master copy of the desktop
661 image for all operations.
662 </item>
663 <item>
664 <p>
665 Since OpenGL is not implemented everywhere, and may not
666 provide hardware-assisted performance in every
667 implementation, an application that requires scaling
668 should also fall back to using the X Video Extension.
669 <p>
670 This project would add support for the X Video Extension
671 to DMX and would add support to VNC to take advantage of
672 this extension without introducing artifacts. This
674 engine to cache a master copy of the desktop image for
675 all operations. This project should also add support
676 for the RGB format to at least one XFree86 driver (e.g.,
677 ATI Radeon).
678 <p>
679 The X Video Extension is one of the few popular
680 extensions that DMX does not support. We recommend
681 implementing the X Video Extension even if scaling is
682 the specific goal of that work.
683 </item>
684 </enum>
685 </p>
686 <p>
688 XCreateScaledWindow extension because of the complexity
690 XPutImageScaled extension because it requires the same amount
691 of work as the XCopyAreaScaled extension, but provides less
692 functionality. Further, server-side scaling with a large
694 performance implications.
695 </p>
696 <p>
697 The back-end scaling, especially with disconnect/reconnect
698 support should be explored in the future after
699 disconnect/reconnect is implemented, but not at the present
700 time.
701 </p>
702 </sect>
704 </article>
705 <!-- Local Variables: -->
706 <!-- fill-column: 72 -->
707 <!-- End: -->