Abstract:Text-based image editing is popular in multimedia and is of great application value, which is also a challenging task as the source image is edited on the basis of a given text, and there is a large cross-modal difference between the image and text. The existing methods can hardly achieve effective direct control and correction of the editing process, but image editing is user preference-oriented, and some editing modules can be bypassed or enhanced by controllability improvement to obtain the results of user preference. Therefore, this study proposes a novel autoencoder-based image editing model according to text descriptions. In this model, an autoencoder is first introduced in stacked generative adversarial networks (SGANs) to provide convenient and direct interactive configuration and editing interfaces. The autoencoder can transform high-dimension feature space between multiple layers into color space and directly correct the intermediate editing results under the color space. Then, a symmetrical detail correction module is constructed to enhance the detail of the edited image and improve controllability, which takes the source image and the edited image as symmetrical exchangeable input to correct the previously input edited image by the fusion of text features. Experiments on the MS-COCO and CUB200 datasets demonstrate that the proposed model can effectively and automatically edit images on the basis of linguistic descriptions while providing user-friendly and convenient corrections to the editing.

Abstract:Zero-shot sketch-based image retrieval uses sketches of unseen classes as query samples for retrieving images of those classes. This task is thus faced with two challenges: the modal gap between a sketch and the image and inconsistencies between seen and unseen classes. Previous approaches tried to eliminate the modal gap by projecting the sketch and the image into a common space and bridge the semantic inconsistencies between seen and unseen classes with semantic embeddings (e.g., word vectors and word similarity). This study proposes a cross-modal self-distillation approach to investigate generalizable features from the perspective of knowledge distillation without the involvement of semantic embeddings in training. Specifically, the knowledge of the pre-trained image recognition network is transferred to the student network through traditional knowledge distillation. Then, according to the cross-modal correlation between a sketch and the image, cross-modal self-distillation indirectly transfers the above knowledge to the recognition of the sketch modality to enhance the discriminative and generalizable features of sketch features. To further promote the integration and propagation of the knowledge within the sketch modality, this study proposes sketch self-distillation. By learning discriminative and generalizable features from the data, the student network eliminates the modal gap and semantic inconsistencies. Extensive experiments conducted on three benchmark datasets, namely Sketchy, TU-Berlin, and QuickDraw, demonstrate the superiority of the proposed cross-modal self-distillation approach to the state-of-the-art ones.

Abstract:The encoder-decoder network based on U-Net and its variants have achieved excellent performance in semantic segmentation of medical images. However, some spatial details are lost during feature extraction, which affects the accuracy of segmentation, and the generalization ability and robustness of these models are unsatisfactory. Therefore, this study proposes a deep convolutional encoder-decoder network with saliency guidance and uncertainty supervision to solve the semantic segmentation problem in multimodal medical images. In this method, the initially generated saliency map and the uncertainty probability map are used as the supervised information to optimize the parameters of the semantic segmentation network. Specifically, the saliency map is generated by the saliency detection network to preliminarily locate the target region in an image, and on this basis, the set of pixel points with uncertain classification is calculated to generate the uncertainty probability map. Then, the two maps are sent into the multi-scale feature fusion network together with the original image to guide the network to focus on the learning of the features in the target region and to enhance the representational capacity of regions with uncertain classification and complex boundaries. In this way, the segmentation performance of the network can be improved. The experimental results reveal that the proposed method can capture more semantic information and outperforms existing semantic segmentation methods in semantic segmentation of multimodal medical images, with strong generalization capability and robustness.

Abstract:A TV logo represents important semantic information of videos. However, its detection and recognition are faced with many problems, including varied categories, complex structures, limited areas, low information content, and severe background disturbance. To improve the generalization ability of the detection model, this study proposes synthesizing TV logo data to construct a training dataset by superimposing TV logo images on background images. Further, a two-stage scalable logo detection and recognition (SLDR) method is put forward, which uses the batch-hard metric learning method to rapidly train the matching model and determine the category of TV logos. In addition, the detection targets can be expanded to unknown categories due to the separation mechanism of detection and recognition in SLDR. The experimental results reveal that synthetic data can effectively improve the generalization ability and detection precision of models, and the SLDR method can achieve comparable precision with the end-to-end model without updating the detection model.

Abstract:Video summarization is an indispensable and critical task in computer vision, the goal of which is to generate a concise and complete video summary by selecting the most informative part of a video. A generated video summary is a set of representative video frames (such as video keyframes) or a short video formed by stitching key video segments in time sequence. Although the study on video summarization has made considerable progress, the existing methods have the problems of deficient temporal information and incomplete feature representation, which can easily affect the correctness and completeness of a video summary. To solve the problems, this study proposes a model based on a spatiotemporal transform network, which includes three modules, i.e., the embedding layer, the feature transformation and fusion layer, and the output layer. Specifically, the embedding layer can simultaneously embed spatial and temporal features, and the feature transformation and fusion layer can realize the transformation and fusion of multi-modal features; finally, the output layer generates the video summary by segment prediction and key shot selection. The spatial and temporal features are embedded separately to fix the problem of deficient temporal information in existing models, and the transformation and fusion of multi-modal features can solve the problem of incomplete feature representation. Sufficient experiments and analyses on two benchmark datasets are conducted, and the results verify the effectiveness of the proposed model.

Abstract:Image captioning is of great theoretical significance and application value, which has attracted wide attention in computer vision and natural language processing. The existing attention mechanism-based image captioning methods integrate the current word and visual cues at the same moment to generate the target word, but they neglect the visual relevance and contextual information, which results in a difference between the generated caption and the ground truth. To address this problem, this paper presents the visual relevance and context dual attention (VRCDA) method. The visual relevance attention incorporates the attention vector of the previous moment into the traditional visual attention to ensure visual relevance, and the context attention is used to obtain much complete semantic information from the global context for better use of the context. In this way, the final image caption is generated via visual relevance and context information. The experiments on the MSCOCO and Flickr30k benchmark datasets demonstrate that VRCDA can effectively describe the image semantics, and compared with several state-of-the-art methods of image captioning, VRCDA can yield superior performance in all evaluation metrics.

Abstract:Graphical bandit is an important model for sequential decision making under uncertainty and has been applied in various real-world scenarios such as social network, electronic commerce, and recommendation system. Existing work on graphical bandits only investigates how to identify the best arm rapidly so as to minimize the cumulative regret while ignoring the privacy protection issue arising in many real-world applications. To overcome this deficiency, a differentially private algorithm is proposed, termed as graph-based arm elimination with differential privacy (GAP), for graphical bandits. On the one hand, GAP updates the arm selection strategy based on empirical mean rewards of arms in an epoch manner. The empirical mean rewards are perturbed by Laplace noise, which makes it hard for malicious attackers to infer rewards of arms from the output of the algorithm, and thus protects the privacy. On the other hand, in each epoch, GAP carefully constructs an independent set of the feedback graph and only explores arms in the independent set, which effectively utilize the information in the graph feedback. It is proved that GAP is differential private and its regret bound matches the theoretical lower bound. Experimental results on synthetic datasets demonstrate that GAP can effectively protect the privacy and achieve cumulative regret comparable to that of existing non-private graphical bandits algorithm.

Abstract:Functional verification is a basic step in VLSI design. With the popularity and development of VLSI, the feasibility and efficiency of functional verification of the whole circuit on a single processor are greatly deficient. The functional verification based on hardware accelerator divides the whole circuit into several smaller sub circuits. When the parallelism of circuit partitioning is better, the time cycle of function verification can be accelerated. Similar to other partitioning problems in circuit design, the circuit partitioning problem for hardware accelerated function verification can be abstracted into graph partitioning problem. In order to meet the requirements of hardware accelerated functional verification, an effective algorithm based on traditional multi-level graph partition strategy is proposed. The algorithm combines the idea of scheduling, and uses the critical path information and timing information of the circuit. The problem of hardware accelerated function verification is transformed into the problem of multi-level partition of directed acyclic graph. The experimental results of random circuit netlist data show that the proposed algorithm can effectively reduce the critical path length and does not cause the growth and deterioration of the number of cut edges.

Abstract:Binary program analysis techniques are widely applied in software security testing, malware analysis and detection, etc. Dynamic analysis is an important analysis method that can accurately show the running status of programs. However, it is confronted with some challenges, such as too high load during target program running and difficulty in dissecting the data structure information in detail. This study proposes a new data flow analysis method based on progressive expansion for binary programs. By taking full advantage of the ability of online data flow analysis, it focuses on the fine-grained analysis for partial program and expands the analysis range progressively to cover the entire program. The method utilizes a divide-and-conquer strategy that can reduce the performance impact on the runtime of the target program and thereby enable the execution of the target code segment sensitive to delay. Meanwhile, this study also presents a correlation analysis method for function parameters based on the memory reference relationship. It can detect the data flow propagation at the function call level and aid in the recovery of the internal data structures of parameters. In the end, this study shows the results of the experiments on the programs in the real environment, which suggest the feasibility and effectiveness of the proposed method. This method does not introduce significant extra analysis overhead while reducing the performance impact on the target program, capable of being applied in binary program analyses in practice.

Abstract:As an important way to access and use web services, REST API provides a technical means for developing and implementing service-oriented architecture-based application systems. However, REST API's design quality varies, so practical and reasonable design guidelines are essential for standardizing and improving REST API design quality. First of all, based on the connotation of REST API, a multi-dimensional, two-layered REST API design guideline classification framework REST API design rule catalog (RADRC) is established. Twenty-five popular design guidelines are classified based on RADRC. Secondly, a REST API design guideline compliance inspection tool, namely RESTer, is implemented. Finally, RESTer is employed to conduct an empirical study on current REST API design by analyzing nearly 2 000 real-world REST API documents from APIs.guru. RESTer analyzes the documents and extracts REST API design information for characterizing REST API design and inspecting compliance with the design guidelines. The empirical study finds that REST APIs of different application categories vary in resources and operation modes, making different categories REST APIs have the characteristics in terms of design guidelines and overall architecture. The empirical study results help understand the characteristics, status quo, and shortcomings of current REST APIs and their adoptions of design guidelines, which is practically significant to improve REST API design quality and design guidelines.

Abstract:Path testing is a very important and widely used structural testing method. Existing path generation methods are either time-consuming or labor-intensive, or they can generate a large number of redundant paths. To solve the above problem, this work mainly studies the optimization model of path selection problem and its evolutionary solution method. The purpose is to reduce the number of redundant paths and reduce test consumption without reducing test coverage. First, a number of paths are selected as the decision variable, and the number of edges and paths included in these paths are taken as the objective to formulate a multi-objective optimization model. Then, the multi-objective evolutionary algorithm is employed to solve the formulated model with the purpose of obtaining the target path set. The proposed method is applied to test 7 benchmark programs and it is compared with the existing method and greedy algorithm. Experimental results show that, compared with other algorithms, the proposed method can reduce the test consumption under the condition of ensuring test sufficiency, thereby improving the test efficiency.

Abstract:As the privacy-preserving problem gains increasing concerns in today's machine learning (ML) world, constructing an ML serving system with a data security guarantee becomes very important. Meanwhile, trusted execution environments (e.g., Intel SGX) have been widely used for developing trusted applications and systems. For instance, Intel SGX offers developers hardware-based secure containers (i.e., enclaves) to guarantee application confidentiality and integrity. This paper presents S3ML, an SGX-based secure serving system for ML inference. S3ML leverages Intel SGX to host ML models for users' privacy protection. To build a practical secure serving system, S3ML addresses several challenges to run model servers inside SGX enclaves. In order to ensure availability and scalability, a frontend ML inference service typically consists of many backend model server instances. When these instances are running inside SGX enclaves, new system architectures and protocols are in need to synchronize cryptographic certificates and keys to construct distributed secure enclave clusters. A dedicated module is designed, it is called attestation-based enclave configuration service in S3ML, responsible for generating, persisting, and distributing certificates and keys among clients and model server instances. The existing infrastructure can then be reused to do transparent load balancing and failover to ensure service high-availability and scalability. Besides, SGX enclaves rely on a special memory region called the enclave page cache (EPC), which has a limited size and is contended by a host’s all enclaves. Therefore, the performance of SGX-based applications is vulnerable to EPC interferences. To satisfy the service-level objective (SLO) of ML inference services, S3ML first integrates lightweight ML framework/models to reduce EPC consumption. Furthermore, through offline analysis, it is found feasible to use EPC paging throughput as indirect monitoring metric to satisfy SLO. Based on this result, S3ML uses real-time EPC paging information to control service load balancing and scaling activities for SLO satisfaction. S3ML has been implemented based on Kubernetes, TensorFlow Lite, and Occlum. The system overhead, feasibility, and effectiveness of S3ML are demonstrated through extensive experiments on a series of popular ML models.

Abstract:Knowledge graph embedding aims to embed entities and relations into a low-dimensional continuous vector space. Due to the data sparsity of knowledge graphs, the performance of knowledge graph embedding is poor in vector representation. Since the type information of entities encompasses rich semantic information, it is introduced to improve the performance. However, the existing methods either do not support the hierarchical structure of type information or the type constraint of relations or complicate the model of the hierarchical structure. This study proposes a novel knowledge graph embedding method combining with hierarchical type information. Specifically, types are embedded into different vector spaces and the hierarchical structure of types is modeled by the partial order relation. Moreover, the vector representations of entities are mapped into the type vector space so that entities and their types can be required to satisfy the partial order relation. The entities and their type constraint of relations in triples are also made to satisfy the partial order relation. Finally, experimental results of link prediction, triple classification and entity typing task on four datasets show that the proposed method outperforms the state-of-the-art baseline methods in vector representation performance.

Abstract:At present, mixed reality (MR) technology is gaining increasingly attention in digital medicine. Targeted at MR of glioma medical image analysis, this study proposes an MR glioma location and regional segmentation algorithm based on the 3D UNet deep learning model, and uses the surface rendering method to render and optimize multi-structure tissue of the glioma image in three-dimensional space. On this basis, three-dimensional registration tracking and visual space sharing algorithms are presented using the interactive markerless and the marker-based graphs for mobile MR to achieve the real-time third-view space sharing for MR multi-devices. In addition, an MR experimental system is designed and implemented for glioma medical image analysis. The experimental results show that the methods proposed in this paper can effectively realize the detection, segmentation and three-dimensional reconstruction of the brain glioma. Through the real-time sharing of mobile MR devices, the proposed methods can effectively achieve MR analysis of glioma medical images to support the auxiliary diagnosis and treatment of glioma, and it also can provide new methods for preoperative planning, medical education and training, etc.

Abstract:Knowledge graphs (KGs) serve as a kind of knowledge base by storing facts with network structure, representing each piece of fact as a triple, i.e. (head, relation, tail). Thanks to the general applications of KGs in various of fields, the embedding learning of knowledge graph has also quickly gained massive attention. This study tries to classify the existing embedding algorithms as five types: translation-based models, tensor factorization-based models, traditional deep learning-based models, graph neural network-based models, and models by fusing extra information. Then, the key ideas, algorithm features, advantages and disadvantages of different embedding models are introduced and analyzed to give the first-time researchers a guideline that can be referenced to help researchers quickly get started.

Abstract:The object detection model has been widely applied in many fields; however, as a machine learning model, it remains a black box to humans. Interpreting the model is conducive to a better understanding of the model and can help judge whether the model is reliable. In view of the interpretability problem of the object detection model, this study proposes that the output of the model should be changed into a specific regression problem that focuses on the existence possibility of the objects of each class. On this basis, the methods to analyze the decision basis and reliability of the object detection model are put forward. Due to the poor versatility of the original image segmentation method, LIME generates unfaithful and ineffective interpretations when interpreting the object detection model. Therefore, the image segmentation method with LIME replaced by DeepLab is put forward and improved, and the improved method can interpret the object detection model. The experiment results prove the superiority of the improved method in interpreting the object detection model.

Abstract:Software testing is a commonly used software quality assurance technique. Mutation testing is a fault-based software testing technique that is widely applied to evaluate the sufficiency of test suites and the effectiveness of software testing techniques. However, the cost of mutation testing is extremely high due to the large number of mutants. This study proposes a mutant reduction technique, DFSampling, guided by data flow analysis and designs three heuristic rules. The random selection technique and the path-aware mutant reduction technique (PAMR) are improved in line with these rules. An empirical study is conducted to evaluate the effectiveness of DFSampling and compare DFSampling with the random selection technique and the PAMR technique in terms of effectiveness. The experimental results show that DFSampling is an effective mutant reduction strategy, which can increase the efficiency of mutation testing.

Abstract:Digital watermarking to form fingerprints in databases is an important approach for database right protection and ownership identification. It provides protection for the sharing and fusion of data. As existing database fingerprinting methods have a deficiency in the universality of data, this study proposes a database fingerprinting approach based on statistical features. This approach first divides the host data into several subsets by an iterative hash function. Then, the statistical feature of each subset is maximized/minimized by an optimization algorithm after extreme values are filtered out. Finally, the optimum threshold is taken as fingerprint information which is calculated by Bayesian decision for minimum errors. This study also theoretically verifies the feasibility and effectiveness of the proposed method. The experimental results on real datasets demonstrate that the method has advantages in both robustness and universality.

Abstract:With the development of the Internet of Things (IoT) technology, the current amount of data generated by the IoT system is increasing, and the data is continuously transmitted to the data center. The traditional IoT data processing and analysis system is inefficient and cannot handle such a large number of data streams. In addition, IoT smart devices have a resource-limited feature, which cannot be ignored during data analysis. This study proposes a new architecture ApproxECIoT (approximate edge computing IoT) suitable for real-time data stream processing of the IoT. It realizes a self-adjusting stratified sampling algorithm to process real-time data streams. The method adjusts the size of the sample strata according to the variance of each stratum while maintaining the given memory budget. This is beneficial to improving the accuracy of the calculation results when resources are limited. Finally, the experimental analysis is performed using simulated datasets and real-world datasets. The results show that ApproxECIoT can still obtain high-accuracy calculation results even with limited resources of the edge nodes.

Abstract:Privacy negotiation performs a pre-protective role against privacy disclosure as it can assist social network users to build a consensus on privacy protection before information sharing. Accountability is an attribute that a subject is responsible for an action or consequence, and it is an important aspect of transparent and explainable artificial intelligence applications. Accountability in the privacy negotiation process in social networks is of great significance for improving the transparency and explainability of application platforms or systems. Although Kekulluoglu et al. proposed an agent-based reciprocal privacy negotiation system, the accountability for the behaviors of agents was not discussed. For this reason, a novel system for agent behavior accountability during privacy negotiation in social networks is designed and implemented, and qualitative and quantitative accountability methods are developed. Moreover, requirements and behavior indicators are also proposed to achieve accountability. Specifically, the qualitative accountability method can accurately determine whether a privacy negotiation agent has misbehavior and pinpoint the specific location of the misbehavior. The quantitative accountability methods include simple quantification, weighted Mahalanobis distance, and improved Minhash and can quantify the severity of the agent’s misbehavior. The experimental data demonstrate the validity and rationality of the proposed system and methods.

Abstract:Adaptive image steganography has been becoming a hot topic, as it conceals covert information within the texture region of an image by employing a defined distortion function, which guarantees remarkable security. In spatial gray-scale image steganography, the research on designing steganographic distortion functions using generative adversarial networks has achieved a significant breakthrough recently. However, related studies of spatial color image steganography have not been reported yet so far. Compared with the gray-scale image steganography, color image steganography should preserve the RGB channel correlation and reasonably assign the embedding capacity among RGB channels simultaneously. This study first proposes a framework based on a generative adversarial network to automatically learn to generate the steganographic distortion function for spatial color images, which is termed CIS-GAN (color image steganography based on generative adversarial network). The generator is composed of two U-Net subnetworks. One of them generates the modification probability matrix, while the other adjusts the positive/negative modification probability to effectively weaken the damage to the RGB channel correlation. The analyzer of gray-scale image steganography is modified as an adversarial part of the network for color images. In addition, the generator can automatically learn to allocate the embedding capacity for the three channels via controlling the total steganographic capacity in the generator’s loss function. The experimental results show that the proposed framework outperforms the advanced steganographic schemes for spatial color images in resisting color image steganalysis.

Abstract:Point-in-spherical-polygon tests are highly required in global data processing. For this reason, this study proposes an adaptive hexagonal hierarchical grid, which overcomes the difficulty of existing hexagonal hierarchical grids in adaptively subdividing grid cells, and applies it to point-in-spherical-polygon tests. First, the initial spherical hexagonal grid is built by uniformly partitioning a sphere using a regular icosahedron. Then, hierarchical grids are constructed by adaptively subdividing hexagonal cells according to whether a grid contains many polygon edges. As a result, the cells not subdivided contain no or only a few edges, called leaf cells. Finally, pre-computing is performed to determine the location attributes (inside/outside the polygon) of such cells or their center points. In the hierarchical structures, the topologies of related points, edges and faces between adjacent hexagonal grid levels are recorded, by which the leaf cells can be quickly located. For a test point, the leaf cell containing it is found quickly, and then whether it is located in the polygon is determined according to the local situation of the cell. Experimental results show that the proposed method has more stable and efficient performance than the existing methods.

Abstract:Recently, the convolutional neural network (CNN) based single-image dynamic scene blind deblurring (SIDSBD) methods have made significant progress. Their success mainly stems from the multi-scale/multi-patch model and the design of the encoder-decoder architecture and the residual block structure. In this paper, a novel multi-scale CNN (MSCNN) is proposed to further exploit the advantages of the multi-scale model, the encoder-decoder architecture, and the residual block structure, which can achieve higher-quality SIDSBD. First, inspired by the spatial pyramid pooling (SPP) and the multi-patch model, this study put forward a hierarchical multi-patch channel attention (HMPCA) strategy to perform adaptive weight assignment for feature images channel-wise by using the global and local feature statistics. The proposed HMPCA uses local information, which can be considered to enlarge the receptive field in the channel direction and thus can enhance the representational ability of the network. Then, different from existing multi-scale models, a novel multi-scale model is built, in which each scale consists of multiple encoders and decoders. Because of the HMPCA, the encoders and decoders at the same scale are not exactly the same. The proposed multi-scale model can be regarded to increase the depth of the encoder-decoder architecture, thus able to improve the deblurring performance of each scale and finally achieve higher-quality blind deblurring for dynamic scenes. Extensive experiments comparing the proposed SIDSBD method with state-of-the-art ones demonstrate the superiority of the method in terms of both qualitative evaluation and quantitative metrics.

Abstract:Regarding the practical problems of the real-time task scheduling and analysis in safety-critical systems such as spacecraft, this study proposes a schedulability determination method based on virtual zoom out period, constructing a strong-hard task (SHT) model to accurately describe real-time tasks, and allocates priority based on task’s time characteristics. Virtualizing all strong real-time task as a hard real-time task, virtually reduces the period of the hard real-time task and calculates the worst virtual execution time, and then determines the schedulability according to the RMS schedulability judgment formula. This paper presents a rigorous proof of the method, which can make a fast schedulability determination on an SHT task set containing n tasks, and the time complexity of this algorithm is only O(n²). Comparative verification was carried out on the China space station computer, and the experiments show that the schedulability determination efficiency is better than the existing methods. The average running time overhead is reduced by 41.8%, and the schedulable ratio is increased by 5.7%.